Issue push Docker image - K3S hosted Pulp

Support
#1

Hello,
I’m testing the Pulp solution to host Docker images and Maven packages, but I am stuck on a problem.
Thanks for your help in advance.

Problem:
When I try to push any Docker image to my registry in order to create a push type container repository, after a few retry of the layer push, I get an Internal Server Error 500.

Expected outcome:
I expect the images to be pushed in a push type container repository and be accessible to pull.

Pulpcore version:
3.68.0

Pulp plugins installed and their versions:
Ansible : 0.23.0
Container : 2.22.0
Deb : 3.4.0
Maven : 0.8.1
Ostree : 2.4.4
Python : 3.12.5
RPM : 3.27.2
Certguard : 3.68.0
File : 3.68.0

Operating system - distribution and version:
My instance is hosted in a K3S cluster on Hetzner Cloud, here are the OS info of the nodes:

Kernel Version: 6.11.8-1-default
OS Image: openSUSE MicroOS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.7.22-k3s1
Kubelet Version: v1.30.6+k3s1

Other relevant data:
The command and the output:

sudo docker push my.artifactory.com/hello-world:latest

The push refers to repository my.artifactory.com/hello-world]
ac28800ec8bb: Pushing [==================================================>] 14.85kB

[ … multiple attemps of pushing … ]
received unexpected HTTP status: 500 Internal Server Error

Logs of pulp-api container 
pulp [6a374f3c1a0a44cf897b431da901f8eb]: django.request:WARNING: Unauthorized: /v2/
('pulp [6a374f3c1a0a44cf897b431da901f8eb]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:02 +0000] "GET /v2/ HTTP/1.1" 401 106 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [02df59e3a36e46239c421f1bd499354b]: ::ffff:10.42.0.6 - admin [02/Dec/2024:14:07:02 +0000] "GET /token/?account=admin&scope=repository%3Ahello-world%3Apush%2Cpull&service=my.artifactory.com HTTP/1.1" 200 1199 "http://my.artifactory.com/token/?account=admin&scope=repository%3Ahello-world%3Apush%2Cpull&service=my.artifactory.com" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [83a3d3373ad049ca8e1d2ea86d032e2c]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:03 +0000] "HEAD /v2/hello-world/blobs/sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 302 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [8bb573effc874fca846e3583c762ad97]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:03 +0000] "POST /v2/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [040fe6c9def04cbdbaa8f37dc7f9a711]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:03 +0000] "PATCH /v2/hello-world/blobs/uploads/019387b2-3518-7501-a6ba-a3719eec6a24 HTTP/1.1" 204 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [126106f7a81c4348b80024c0cdff9061]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:03 +0000] "PUT /v2/hello-world/blobs/uploads/019387b2-3518-7501-a6ba-a3719eec6a24?digest=sha256%3Ac1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 201 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [d77ba4056d734467b6f7d215293954c2]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:03 +0000] "HEAD /v2/hello-world/blobs/sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 302 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [9781575f50404a6d80f639f90342a8cc]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:09 +0000] "POST /v2/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [772f2510b91c4568885b9ab8e6fdf7b5]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:09 +0000] "PATCH /v2/hello-world/blobs/uploads/019387b2-4c3e-797b-9ce7-8678d0145a86 HTTP/1.1" 204 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [83ab64dc58ff4293b71dae51f3eeb5b5]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:09 +0000] "PUT /v2/hello-world/blobs/uploads/019387b2-4c3e-797b-9ce7-8678d0145a86?digest=sha256%3Ac1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 201 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [b4b4fe8d855f43759a5c20c52c0ab176]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:09 +0000] "HEAD /v2/hello-world/blobs/sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 302 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [e91679a691354641afabcca94eefcec7]: ::ffff:10.42.1.1 - - [02/Dec/2024:14:07:10 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3136 "-" "kube-probe/1.30"',)
('pulp [181cd4441683447db486180c8b9e8045]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:20 +0000] "POST /v2/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [07945e79092448daa3188abad32aed72]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:20 +0000] "PATCH /v2/hello-world/blobs/uploads/019387b2-7711-7f64-a2aa-5438e6d52fd5 HTTP/1.1" 204 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [2e6b8d7ca0a946f0b9574a5139fa711d]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:20 +0000] "PUT /v2/hello-world/blobs/uploads/019387b2-7711-7f64-a2aa-5438e6d52fd5?digest=sha256%3Ac1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 201 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [deff7b161dc74869a98d3b0c6d5a91e2]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:20 +0000] "HEAD /v2/hello-world/blobs/sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 302 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [a6ead7f5790c4ea99e1ce29534da3e85]: ::ffff:10.42.1.1 - - [02/Dec/2024:14:07:30 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3136 "-" "kube-probe/1.30"',)
('pulp [7a9828717ce4403a82193cf47b85c864]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:36 +0000] "POST /v2/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [3c1d196af3ed451ebf6842422e4d3c68]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:36 +0000] "PATCH /v2/hello-world/blobs/uploads/019387b2-b5c1-7e07-85ff-3cbf5e94516c HTTP/1.1" 204 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [6247e063c3b443aa8e4b00b52e5feafe]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:36 +0000] "PUT /v2/hello-world/blobs/uploads/019387b2-b5c1-7e07-85ff-3cbf5e94516c?digest=sha256%3Ac1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 201 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [2bee40f0a5b84cc08d32630ff69778f7]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:36 +0000] "HEAD /v2/hello-world/blobs/sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 302 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [046c51af34b7462bb1bcb92170a681cd]: ::ffff:10.42.1.1 - - [02/Dec/2024:14:07:50 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3136 "-" "kube-probe/1.30"',)
('pulp [879f33bbdde449b0b9739ffca8743698]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:57 +0000] "POST /v2/hello-world/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [a1958538d0314701a09c61ce45571e7d]: ::ffff:10.42.0.15 - - [02/Dec/2024:14:07:57 +0000] "PATCH /v2/hello-world/blobs/uploads/019387b3-07dd-75cc-b753-a69d2763249f HTTP/1.1" 204 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [445246bc06ce456aa5ef7b340f1f12b7]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:57 +0000] "PUT /v2/hello-world/blobs/uploads/019387b3-07dd-75cc-b753-a69d2763249f?digest=sha256%3Ac1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 201 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
('pulp [c91ebc8a137742b7b053a3e31bad6b8a]: ::ffff:10.42.0.6 - - [02/Dec/2024:14:07:58 +0000] "HEAD /v2/hello-world/blobs/sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e HTTP/1.1" 302 0 "-" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"',)
Logs of pulp-content container (multiple times because of multiple retries probably): 
[2024-12-02 14:07:03 +0000] [22] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py", line 559, in _handle
    return await handler(request)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulp_container/app/registry.py", line 340, in get_by_digest
    return await Registry._dispatch(artifact, headers)
  File "/usr/local/lib/python3.9/site-packages/pulp_container/app/registry.py", line 90, in _dispatch
    raise Exception("Expected path '{}' is not found".format(path))
Exception: Expected path '/var/lib/pulp/media/artifact/c1/ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e' is not found
::ffff:10.42.0.6 [02/Dec/2024:14:07:03 +0000] "HEAD /pulp/container/hello-world/blobs/sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e?expires=1733152023&validate_token=ed13eebae99dfcb5662ee2a9a7e40d925657ab22938e25fd0c63c81435e39dee:de250b6ecc89bae64eef14a5b1251b376a41ee3a7fac825b09628bfb33bd466b HTTP/1.1" 500 191 "http://my.artifactory.com/pulp/container/hello-world/blobs/sha256:c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e?expires=1733152023&validate_token=ed13eebae99dfcb5662ee2a9a7e40d925657ab22938e25fd0c63c81435e39dee:de250b6ecc89bae64eef14a5b1251b376a41ee3a7fac825b09628bfb33bd466b" "docker/27.2.0 go/go1.21.13 git-commit/3ab5c7d kernel/6.8.0-1018-azure os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \(linux\))"
#2

If it is possible can you post your benchmark script? How many pushes are you doing? Are they in parallel?

#3

Here is the step by step I’m doing to get to the error:

  • Clone pulp repo from GitHub and do make deploy from 1.0.0-beta.4 branch

  • Deploy the Pulp instance using the yaml file below

Pulp instance config 
apiVersion: repo-manager.pulpproject.org/v1beta2
kind: Pulp
metadata:
  name: pulp
  namespace: pulp
spec:
  deployment_type: pulp
  image_version: stable
  image_web_version: stable
  api:
    replicas: 1
  content:
    replicas: 1
  worker:
    replicas: 1

  pulp_settings:
    content_origin: http://my.artifactory.com
    api_root: "/pulp/"
    token_server: http://my.artifactory.com/token/
    allowed_export_paths:
      - /tmp
    allowed_import_paths:
      - /tmp
    telemetry: false
  • Then I set up a custom Ingress based on the one which is created when you ask for ingress type in pulp_settings. I do that because I want to use letsencrypt for TLS.
    Is it a problem ? I didn’t find how to use the native ingress with letsencrypt.
ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: 120s
    nginx.ingress.kubernetes.io/proxy-read-timeout: 120s
    nginx.ingress.kubernetes.io/proxy-send-timeout: 120s
    nginx.org/client-max-body-size: 10m
    web: "false"
  labels:
    app.kubernetes.io/managed-by: pulp-operator
    app.kubernetes.io/part-of: pulp
    pulp_cr: pulp
  name: pulp
  namespace: pulp
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - my.artifactory.com
    secretName: pulp-tls
  rules:
  - host: my.artifactory.com
    http:
      paths:
      - backend:
          service:
            name: pulp-content-svc
            port:
              name: content-24816
        path: /pulp/content/
        pathType: Prefix
      - backend:
          service:
            name: pulp-api-svc
            port:
              name: api-24817
        path: /pulp/api/v3/
        pathType: Prefix
      - backend:
          service:
            name: pulp-api-svc
            port:
              name: api-24817
        path: /auth/login/
        pathType: Prefix
      - backend:
          service:
            name: pulp-api-svc
            port:
              name: api-24817
        path: /
        pathType: Prefix
      - backend:
          service:
            name: pulp-api-svc
            port:
              name: api-24817
        path: /pulp_ansible/galaxy/
        pathType: Prefix
      - backend:
          service:
            name: pulp-api-svc
            port:
              name: api-24817
        path: /v2/
        pathType: Prefix
      - backend:
          service:
            name: pulp-api-svc
            port:
              name: api-24817
        path: /extensions/v2/
        pathType: Prefix
      - backend:
          service:
            name: pulp-content-svc
            port:
              name: content-24816
        path: /pulp/container/
        pathType: Prefix
      - backend:
          service:
            name: pulp-api-svc
            port:
              name: api-24817
        path: /token/
        pathType: Prefix
      - backend:
          service:
            name: pulp-api-svc
            port:
              name: api-24817
        path: /pypi/
        pathType: Prefix

And of course the DNS is setup so that the domain my.artifactory.com is pointing to the load balancer IP (no issue on this)

  • Next I retrieve the admin password

  • Finally I log in to Pulp and try to push:
    sudo docker login my.artifactory.com -u admin -p ADMIN_PWD

#4

What I meant by benchmarking is that I am testing Pulp basically !

At the moment all operations are manual.

#5

Thank you for the very detailed information!

From the Pulp CR provided, it seems like you didn’t configure a persistent volume nor an object storage for this installation, which can cause the error we are seeing in pulp-content logs.
This is a known issue with emptyDir configuration:

“Configuring Pulp operator with emptyDir will fail the execution of some plugins. For example, pulp-container plugin needs to access the data created by pulpcore-api component through pulpcore-content pod, but since each pod has its own emptyDir volume - and their data is not shared between them - Pulp will not work as expected.”

we have an issue open to remove the support of installations using emptyDir and avoid such errors: Remove support to `emptyDir` and add a validation for missing storage definition · Issue #1345 · pulp/pulp-operator · GitHub

So, to fix this, you can check the Pulp Operator storage configuration documentation to configure pulp-operator with a k8s persistent volume or an object storage.

2 Likes
#6

Indeed the persistent storage was the issue. That also explains why I had to reinstall the whole Pulp instance the next day because of crashes.

My problems are solved, I was able to push a Docker image to create a push type repository, manage access using accounts, and pull the image.

Thank you very much !

PS: by any chance, do you have an idea on why the authentication using Podman would not work (returning success from podman login even when credentials are intentionally wrong) whereas docker login works normally ?

2 Likes
#7

Glad to know the issue is fixed!

hmm … since docker login works normally, I guess podman is using the credentials from docker:
https://docs.podman.io/en/stable/markdown/podman-login.1.html

“Podman first searches for the username and password in the ${XDG_RUNTIME_DIR}/containers/auth.json, if they are not valid, Podman then uses any existing credentials found in $HOME/.docker/config.json.”

2 Likes