Pulp Container Domain getting 404 on Image Pull

Problem:
Unable to pull container image from pulp when using Domains

Expected outcome:
Able to pull image as expected

Pulpcore version:

  • component: core
    domain_compatible: true
    module: pulpcore.app
    package: pulpcore
    version: 3.74.1

Pulp plugins installed and their versions:

  • component: container
    domain_compatible: true
    module: pulp_container.app
    package: pulp-container
    version: 2.24.1
  • component: rpm
    domain_compatible: true
    module: pulp_rpm.app
    package: pulp-rpm
    version: 3.29.0
  • component: certguard
    domain_compatible: true
    module: pulp_certguard.app
    package: pulpcore
    version: 3.74.1
  • component: file
    domain_compatible: true
    module: pulp_file.app
    package: pulpcore
    version: 3.74.1

Operating system - distribution and version:
RHEL 9 based pulp containers

pulp-api-578b78589b-4f8wf       1/1     Running   0             2d11h
pulp-content-7c579f4748-vpf9f   1/1     Running   0             2d11h
pulp-worker-5bf69d65d9-z2mrw    1/1     Running   0             2d11h

Other relevant data:
Followed instructions from Domain support - Pulp Project

Domain

- description: Nautilus  Container Repo                                           
  hide_guarded_distributions: false                                                         
  name: nautilus                                                                            
  prn: prn:core.domain:01965e79-d2b6-707a-950c-367815ac6622                                 
  pulp_created: '2025-04-22T17:09:24.278948Z'                                               
  pulp_href: /pulp/default/api/v3/domains/01965e79-d2b6-707a-950c-367815ac6622/             
  pulp_labels: {}                                                                           
  pulp_last_updated: '2025-04-25T16:54:21.265977Z'                                          
  redirect_to_object_storage: true                                                          
  storage_class: pulpcore.app.models.storage.FileSystem                                     
  storage_settings:                                                                         
    base_url: /                                                                             
    directory_permissions_mode: null                                                        
    file_permissions_mode: 444                                                              
    hidden_fields: []                                                                       
    location: /var/lib/pulp/nautilus/

Repository

   - description: null
      latest_version_href: /pulp/nautilus/api/v3/repositories/container/container/01965e7b-b585-7471-943f-89288c09b615/versions/0/
      manifest_signing_service: null
      name: linux
      prn: prn:container.containerrepository:01965e7b-b585-7471-943f-89288c09b615
      pulp_created: '2025-04-22T17:11:27.880758Z'
      pulp_href: /pulp/nautilus/api/v3/repositories/container/container/01965e7b-b585-7471-943f-89288c09b615/
      pulp_labels: {}
      pulp_last_updated: '2025-04-22T17:11:27.902092Z'
      remote: null
      retain_repo_versions: null
      versions_href: /pulp/nautilus/api/v3/repositories/container/container/01965e7b-b585-7471-943f-89288c09b615/versions/

Distribution

    - base_path: linux
  content_guard: /pulp/nautilus/api/v3/contentguards/core/content_redirect/0196624d-4971-7455-83c9-6e6181bd6381/
  description: null
  hidden: false
  name: linux
  namespace: /pulp/default/api/v3/pulp_container/namespaces/0196624d-4987-7cff-b357-6ec3631ad89c/
  no_content_change_since: null
  private: false
  prn: prn:container.containerdistribution:01967c1c-4466-7484-aa1a-07c4ef88c9b0
  pulp_created: '2025-04-28T11:15:49.479633Z'
  pulp_href: /pulp/nautilus/api/v3/distributions/container/container/01967c1c-4466-7484-aa1a-07c4ef88c9b0/
  pulp_labels: {}
  pulp_last_updated: '2025-04-28T11:15:49.479650Z'
  registry_path: pulp-east.dev.aws.xyz.net/linux
  remote: null
  repository: /pulp/nautilus/api/v3/repositories/container/container/01965e7b-b585-7471-943f-89288c09b615/
  repository_version: null

In settings

DOMAIN_ENABLED = True
TOKEN_AUTH_DISABLED = True
  1. push image works fine

    podman push pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9:latest
    Getting image source signatures
    Copying blob d2e1db155a8c done |
    Copying config 119e4066ca done |
    Writing manifest to image destination

  2. Search looks good

    podman search pulp-east.dev.aws.xyz.net/nautilus/linux
    NAME DESCRIPTION
    pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9

3, You can get the tags from the api

curl https://pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9/tags/list
{"name":"nautilus/linux/lrh9","tags":["latest"]}
  1. If we try and pull the image we get a 404

    podman pull pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9:latest
    Trying to pull pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9:latest
    Error: initializing source docker://pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9:latest: reading manifest latest in pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9: StatusCode: 404, "\n<!doctype html>\n<html lang=\"en\">\n<head>\n <title>..."

  2. The API logs

Sorry hit enter to soon here are logs

pulp [c18d25da1c844b5da8c1a6d17f0aadb6]: django.request:WARNING: Unauthorized: /v2/

(‘pulp [c18d25da1c844b5da8c1a6d17f0aadb6]: 100.72.17.160 - - [28/Apr/2025:12:20:54 +0000] “GET /v2/ HTTP/1.1” 401 106 “-” “containers/5.32.2 (github.com/containers/image)”’,)
(‘pulp [715fda2ff868485d8d4afb7418563d78]: 100.72.17.160 - admin [28/Apr/2025:12:20:54 +0000] “GET /v2/nautilus/linux/lrh9/manifests/latest HTTP/1.1” 302 0 “-” “containers/5.32.2 (github.com/containers/image)”’,)
pulp [8a963c1793c44451a7d84471ead9e139]: django.request:WARNING: Not Found: /pulp/container/default/nautilus/linux/lrh9/manifests/latest
(‘pulp [8a963c1793c44451a7d84471ead9e139]: 100.72.17.160 - admin [28/Apr/2025:12:20:54 +0000] “GET /pulp/container/default/nautilus/linux/lrh9/manifests/latest?expires=1745844630&validate_token=09c3500c59a343b6aa2c28cc51f14dfa5b56bd9043e8c4de327499750e054f3c:6b6b651b5b0b3b888c9cccd6400b102326cb5cb99fdabdf39d119ae9fe04d33e HTTP/1.1” 404 179 “https://pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9/manifests/latest” “containers/5.32.2 (github.com/containers/image)”’,)

@woolsgrs Full domain support was added in pulp_container 2.25. In 2.24 we added only partial domain support to allow installing the plugin with other domain enabled plugins. The partial support only worked in the default domain, this is why you see default in the redirect link. If you upgrade to the latest then it should work (you might have to delete and recreate the objects first though)

Thanks much,

  1. I deleted the distribution and repository

  2. Updated pulp_container 2.25.0

  3. Re-created the repository/distribution where I can see some changes

  4. Pushed Image

  5. Pull fails with the same 404 albeit I see its not using default

    (‘pulp [61223bdb08fb459ea818657cebe4111e]: 100.72.17.160 - admin [28/Apr/2025:14:58:37 +0000] “POST /v2/nautilus/linux/lrh9/blobs/uploads/ HTTP/1.1” 202 0 “-” “containers/5.32.2 (github.com/containers/image)”’,)
    (‘pulp [d36aaf18aa5744c8ba1ba4368d9fef81]: 100.72.124.185 - admin [28/Apr/2025:14:58:37 +0000] “PATCH /v2/nautilus/linux/lrh9/blobs/uploads/01967ce8-3df1-7d8e-8f97-df1af074e3d4 HTTP/1.1” 204 0 “-” “containers/5.32.2 (github.com/containers/image)”’,)
    (‘pulp [5e08613a81b445dfa3896d2a6f68f154]: 100.72.17.160 - admin [28/Apr/2025:14:58:38 +0000] “PUT /v2/nautilus/linux/lrh9/blobs/uploads/01967ce8-3df1-7d8e-8f97-df1af074e3d4?digest=sha256%3A119e4066ca9837b871a49c0c5136386ba3afeb924615a0bda0cdec7f0a9e872c HTTP/1.1” 201 0 “-” “containers/5.32.2 (github.com/containers/image)”’,)
    pulp [7dbda59e7e194fb6aedd05dd1973c50b]: pulpcore.tasking.tasks:INFO: Starting task 01967ce8-461e-7e28-bd84-fd3579f471c6 in domain: nautilus
    pulp [7dbda59e7e194fb6aedd05dd1973c50b]: pulpcore.tasking.tasks:INFO: Task completed 01967ce8-461e-7e28-bd84-fd3579f471c6 in domain: nautilus
    (‘pulp [7dbda59e7e194fb6aedd05dd1973c50b]: 100.72.124.185 - admin [28/Apr/2025:14:58:39 +0000] “PUT /v2/nautilus/linux/lrh9/manifests/latest HTTP/1.1” 201 0 “-” “containers/5.32.2 (github.com/containers/image)”’,)
    pulp [03b55ba8daa347128e8574561a7ee08e]: django.request:WARNING: Unauthorized: /v2/
    (‘pulp [03b55ba8daa347128e8574561a7ee08e]: 100.72.124.185 - - [28/Apr/2025:14:58:53 +0000] “GET /v2/ HTTP/1.1” 401 106 “-” “containers/5.32.2 (github.com/containers/image)”’,)
    (‘pulp [c4cdb2a29a7b4e0db8da79ba09644de0]: 100.72.17.160 - admin [28/Apr/2025:14:58:53 +0000] “GET /v2/nautilus/inux/lrh9/manifests/latest HTTP/1.1” 302 0 “-” “containers/5.32.2 (github.com/containers/image)”’,)
    pulp [3e319b568bd64421ac13002c4b773b01]: django.request:WARNING: Not Found: /pulp/container/nautilus/linux/lrh9/manifests/latest
    (‘pulp [3e319b568bd64421ac13002c4b773b01]: 100.72.124.185 - admin [28/Apr/2025:14:58:53 +0000] “GET /pulp/container/nautilus/linux/lrh9/manifests/latest?expires=1745855933&validate_token=90dd12dd516ba19c4b66c68dd46819846ced0c09f2fa5fb6a39f675327959231:dbb323275d191826845a9dc230aa70eeeb413177b74131f30c52a643c91a33a1 HTTP/1.1” 404 179 “https://pulp-east.dev.aws.xyx.net/v2/nautilus/linux/lrh9/manifests/latest” “containers/5.32.2 (github.com/containers/image)”’,)

    versions:

    • component: core
      domain_compatible: true
      module: pulpcore.app
      package: pulpcore
      version: 3.74.1
    • component: container
      domain_compatible: true
      module: pulp_container.app
      package: pulp-container
      version: 2.25.0
    • component: rpm
      domain_compatible: true
      module: pulp_rpm.app
      package: pulp-rpm
      version: 3.29.0
    • component: certguard
      domain_compatible: true
      module: pulp_certguard.app
      package: pulpcore
      version: 3.74.1
    • component: file
      domain_compatible: true
      module: pulp_file.app
      package: pulpcore
      version: 3.74.1

Cheers

Sorry silly question, when you upgraded pulp-container did you restart all of Pulp. The content, api and task workers should all be running the same version, you can check with pulp status.

Second can you post the created distribution?

Sure see below for pulp status, rebuilt the container hosting pulp and restarted all three services

content_settings:
  content_origin: https://pulp-east.dev.aws.xyz.net
  content_path_prefix: /pulp/content/
database_connection:
  connected: true
domain_enabled: true
online_api_apps:
- last_heartbeat: '2025-04-29T07:01:20.376316Z'
  name: 18@pulp-api-6c74bb59c8-gn2sm
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:20.384631Z'
  name: 19@pulp-api-6c74bb59c8-gn2sm
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
online_content_apps:
- last_heartbeat: '2025-04-29T07:01:14.069458Z'
  name: 17@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:14.070751Z'
  name: 23@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:14.074383Z'
  name: 21@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:14.400153Z'
  name: 15@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:15.859991Z'
  name: 29@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:16.048399Z'
  name: 27@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:16.233009Z'
  name: 25@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:16.559796Z'
  name: 32@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:16.788277Z'
  name: 30@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
- last_heartbeat: '2025-04-29T07:01:19.649632Z'
  name: 19@pulp-content-65b55fd74-hwxr8
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
online_workers:
- current_task: null
  last_heartbeat: '2025-04-29T07:01:20.062174Z'
  name: 1@pulp-worker-5cdb8cd8bc-n7cjx
  prn: prn:core.worker:01967cdf-d684-70ab-b517-429906b9f557
  pulp_created: '2025-04-28T14:49:26.406889Z'
  pulp_href: /pulp/default/api/v3/workers/01967cdf-d684-70ab-b517-429906b9f557/
  pulp_last_updated: '2025-04-28T14:49:26.406913Z'
  versions:
    certguard: 3.74.1
    container: 2.25.0
    core: 3.74.1
    file: 3.74.1
    rpm: 3.29.0
redis_connection:
  connected: true
storage:
  free: 9223368652922814464
  total: 9223372036853727232
  used: 3383930912768
versions:
- component: core
  domain_compatible: true
  module: pulpcore.app
  package: pulpcore
  version: 3.74.1
- component: container
  domain_compatible: true
  module: pulp_container.app
  package: pulp-container
  version: 2.25.0
- component: rpm
  domain_compatible: true
  module: pulp_rpm.app
  package: pulp-rpm
  version: 3.29.0
- component: certguard
  domain_compatible: true
  module: pulp_certguard.app
  package: pulpcore
  version: 3.74.1
- component: file
  domain_compatible: true
  module: pulp_file.app
  package: pulpcore
  version: 3.74.1

Can you try curling the manifest and seeing what error you get?

curl -H "Accept: application/vnd.docker.distribution.manifest.v2+json" https://pulp-east.dev.aws.xyx.net/v2/nautilus/linux/lrh9/manifests/latest

The correct response should be a 302 to the content app.

Does get a 302 but fails on the re-direct

curl --head -L -H "Accept: application/vnd.docker.distribution.manifest.v2+json" https://pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9/manifests/latest
HTTP/2 302
date: Wed, 30 Apr 2025 08:23:52 GMT
content-type: text/html; charset=utf-8
content-length: 0
location: https://pulp-east.dev.aws.xyz.net/pulp/container/nautilus/linux/lrh9/manifests/latest?expires=1746004838&validate_token=5c04f07b7001b6a2e16629b2d0c734f19bc9ea830051d3e802a9eb04b7363b33:8ad4ff74b0020204f0d4f0c90860ffc20e917729f35c1c6261f33f74b25f5e39
x-pulp-cache: HIT
allow: GET, PUT, HEAD, OPTIONS
docker-distribution-api-version: registry/2.0
x-registry-supports-signatures: 1
x-frame-options: DENY
x-content-type-options: nosniff
referrer-policy: same-origin
cross-origin-opener-policy: same-origin
correlation-id: 7389ffe1395540bd8d8234956d924093
access-control-expose-headers: Correlation-ID
strict-transport-security: max-age=31536000; includeSubDomains

HTTP/2 404
date: Wed, 30 Apr 2025 08:23:52 GMT
content-type: text/html; charset=utf-8
content-length: 179
x-frame-options: DENY
x-content-type-options: nosniff
referrer-policy: same-origin
cross-origin-opener-policy: same-origin
correlation-id: 1390b5e73b7844c988516b35ff0fb762
access-control-expose-headers: Correlation-ID
strict-transport-security: max-age=31536000; includeSubDomains

Looking at the redirect

curl -Ls -o /dev/null -w %{url_effective} -H "Accept: application/vnd.docker.distribution.manifest.v2+json" https:/pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9/manifests/latest
https://pulp-east.dev.aws.xyz.net/pulp/container/nautilus/linux/lrh9/manifests/latest?expires=1746004428&validate_token=29c5ee44553ae4acb9548435b95e83e80df55712061462d127d823e1085da32f:8ce2375b49e110e8159c9d9cb3e8c3ddcf56557950c5d9bf20455c9753a68871

However, I have now noticed the default domain has stopped working which should use the default content_path_prefix /pulp/content above seems to have /pulp/container ?

Do you have a custom install of Pulp? For pulp_container we extend the content app to serve the artifacts from the registry api, so that is why you see pulp/container in the redirect url. You need this route [0] for your reverse proxy.

[0] https://github.com/pulp/pulp_container/blob/main/pulp_container/app/webserver_snippets/nginx.conf#L22-L30

If I update ingress to fwd /pulp/container to the content then we get further

curl --head -L -H "Accept: application/vnd.docker.distribution.manifest.v2+json" https://pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9/manifests/latest
HTTP/2 302
date: Wed, 30 Apr 2025 12:52:00 GMT
content-type: text/html; charset=utf-8
content-length: 0
location: https://pulp-east.dev.aws.xyz.net/pulp/container/nautilus/linux/lrh9/manifests/latest?expires=1746020335&validate_token=e4508d9fb3057a9b54f884dd7c81f16ac5152d27cdf242da977cb8a776a19283:4612528b389ce0da7ede5cf050a922b931148582d6e6218655aa47ca8b3f2639
x-pulp-cache: HIT
allow: GET, PUT, HEAD, OPTIONS
docker-distribution-api-version: registry/2.0
x-registry-supports-signatures: 1
x-frame-options: DENY
x-content-type-options: nosniff
referrer-policy: same-origin
cross-origin-opener-policy: same-origin
correlation-id: 3160c53929e548ee8a1387b2a55b0980
access-control-expose-headers: Correlation-ID
strict-transport-security: max-age=31536000; includeSubDomains

HTTP/2 200
date: Wed, 30 Apr 2025 12:52:00 GMT
content-type: application/vnd.docker.distribution.manifest.v2+json; charset=utf-8
content-length: 429
docker-content-digest: sha256:459f7cfffc717dd3080679f269047695a697d015c15e92852f84ed8f8d5cadc2
x-pulp-cache: HIT
strict-transport-security: max-age=31536000; includeSubDomains

Now I get a 500 this is logged by content app

[30/Apr/2025:16:09:51 +0000] "GET /pulp/container/nautilus/linux/lrh9/manifests/latest?expires=1746032991&validate_token=49eb9d52502a12656073230d4bdd283c82b110f09258a7935fa3564ab4134ec0:96398e5c46de8b5252b73dedf227db29ebc85883f8018772408865c880eb1dcc HTTP/1.1" 200 742 "https://pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9/manifests/latest" "containers/5.32.2 (github.com/containers/image)"
[2025-04-30 16:09:53 +0000] [32] [ERROR] Error handling request from 
Traceback (most recent call last):
  File "/usr/local/lib64/python3.12/site-packages/aiohttp/web_protocol.py", line 480, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.12/site-packages/aiohttp/web_app.py", line 569, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib64/python3.12/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulpcore/content/authentication.py", line 27, in guid
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulpcore/content/authentication.py", line 58, in authenticate
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulpcore/cache/cache.py", line 354, in cached_function
    response = await self.make_entry(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulpcore/cache/cache.py", line 400, in make_entry
    response = await handler(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulp_container/app/registry.py", line 297, in get_by_digest
    return await Registry._dispatch(artifact, headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pulp_container/app/registry.py", line 91, in _dispatch
    raise Exception("Expected path '{}' is not found".format(path))
Exception: Expected path '/var/lib/pulp/media/artifact/01965e79-d2b6-707a-950c-367815ac6622/11/9e4066ca9837b871a49c0c5136386ba3afeb924615a0bda0cdec7f0a9e872c' is not found
 [30/Apr/2025:16:09:52 +0000] "GET /pulp/container/nautilus/linux/lrh9/blobs/sha256:119e4066ca9837b871a49c0c5136386ba3afeb924615a0bda0cdec7f0a9e872c?expires=1746032992&validate_token=b735fc4713b6c68d9978510586c58312c4256c5fcd014834241ee957c8a48e5f:e045fcc6a3d5f2a2e720dae15250d0e0911d89c5779e4adcb94bb2e2321b13e9 HTTP/1.1" 500 247 "https://pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9/blobs/sha256:119e4066ca9837b871a49c0c5136386ba3afeb924615a0bda0cdec7f0a9e872c" "containers/5.32.2 (github.com/containers/image)"

1

I believe this content was created before you upgraded Pulp. Please remove the repository and run orphan cleanup for this domain. The orphan cleanup needs to be run with --protection-time 0.

Thanks tried this but its not removing the original image pushed as when trying to push again I get the same 500 error when its checks to see if the digest exists.

Steps taken

  1. pulp --domain nautilus container distribution destroy --name linux

  2. pulp --domain nautillus container repository destroy --name linux

  3. pulp --domain nautilus orphan cleanup --protection-time 0

    progress_reports:

    • code: clean-up.content
      done: 0
      message: Clean up orphan Content
      state: completed
      suffix: null
      total: 0
    • code: clean-up.artifacts
      done: 12
      message: Clean up orphan Artifacts
      state: completed
      suffix: null
      total: 12

it still left the file system in place.

pulp@container[pulp-content-65b55fd74-hwxr8]:/var/lib/pulp/nautilus/artifact# ls -alRnh
.:
total 12K
drwxr-xr-x.  3 999 999 6.0K Apr 23 10:59 .
drwxr-xr-x.  3 999 999 6.0K Apr 23 10:59 ..
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 01965e79-d2b6-707a-950c-367815ac6622

./01965e79-d2b6-707a-950c-367815ac6622:
total 68K
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 .
drwxr-xr-x.  3 999 999 6.0K Apr 23 10:59 ..
drwxr-xr-x.  2 999 999 6.0K May  1 10:31 0b
drwxr-xr-x.  2 999 999 6.0K Apr 28 14:58 11
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 1a
drwxr-xr-x.  2 999 999 6.0K May  1 10:31 21
drwxr-xr-x.  2 999 999 6.0K May  1 10:30 3d
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 45
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 76
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 91
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 95
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 c7
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 d4
drwxr-xr-x.  2 999 999 6.0K May  1 10:31 d9
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 dc
drwxr-xr-x.  2 999 999 6.0K Apr 28 14:58 e1
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 f6

./01965e79-d2b6-707a-950c-367815ac6622/0b:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:31 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/11:
total 12K
drwxr-xr-x.  2 999 999 6.0K Apr 28 14:58 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..
-rw-rwxr--.  1 999 999 1.5K Apr 28 14:58 9e4066ca9837b871a49c0c5136386ba3afeb924615a0bda0cdec7f0a9e872c

./01965e79-d2b6-707a-950c-367815ac6622/1a:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/21:
total 12K
drwxr-xr-x.  2 999 999 6.0K May  1 10:31 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..
-rw-rwxr--.  1 999 999   32 May  1 10:31 bf5edcaeefec6263e711a1a40bd4c8a50418aad3038fe73e4202b968d6f726

./01965e79-d2b6-707a-950c-367815ac6622/3d:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:30 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/45:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/76:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/91:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/95:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/c7:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/d4:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/d9:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:31 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/dc:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..

./01965e79-d2b6-707a-950c-367815ac6622/e1:
total 56M
drwxr-xr-x.  2 999 999 6.0K Apr 28 14:58 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..
-rw-rwxr--.  1 999 999  56M Apr 28 14:58 7c850423e5d29f7ce27c6b5c59cfc48bf537dd15a31390b843cb94be714828

./01965e79-d2b6-707a-950c-367815ac6622/f6:
total 8.0K
drwxr-xr-x.  2 999 999 6.0K May  1 10:26 .
drwxr-xr-x. 17 999 999 6.0K May  1 10:26 ..
p

I recreated the repository and distribution but still get

podman push lis-pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9:latest
Getting image source signatures
Copying blob d2e1db155a8c done   | 
Copying config 119e4066ca [--------------------------------------] 8.0b / 1.5KiB | 8.7 MiB/s
Error: writing blob: checking whether a blob sha256:119e4066ca9837b871a49c0c5136386ba3afeb924615a0bda0cdec7f0a9e872c exists in pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9: received unexpected HTTP status: 500 Internal Server Error


raise Exception("Expected path '{}' is not found".format(path))                                                                     

Exception: Expected path ‘/var/lib/pulp/media/artifact/01965e79-d2b6-707a-950c-367815ac6622/11/9e4066ca9837b871a49c0c5136386ba3afeb92461
5a0bda0cdec7f0a9e872c’ is not found
100.72.124.185 [01/May/2025:10:27:21 +0000] “HEAD /pulp/container/nautilus/linux/lrh9/blobs/sha256:119e4066ca9837b871a49c0c5136386ba
3afeb924615a0bda0cdec7f0a9e872c?expires=1746097848&validate_token=31e04c253e854d0338b7c9402da5d9f22251d4f496e5064114dd38fce6e9d021:2d904
517ebe559365887ecdd6783064b8ce50253b24059c23486f877a128e0a6 HTTP/1.1” 500 192 “https://pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9/blobs/sha256:119e4066ca9837b871a49c0c5136386ba3afeb924615a0bda0cdec7f0a9e872c” “containers/5.32.2 (github.com/containers/im
age)”

Do I recreate the domain and specify a diff file path? Can I then wipe the files or there will still be references in the DB

Can you delete the repository/distributions and run orphan cleanup in the default domain as well? I wonder why there was zero content deleted. Are there other repositories in the domain you haven’t deleted? Try pulp --domain nautilus container repository -t push list.

It does seem that there are bad content or artifact references in the database that need to be removed. Here are some shell commands to remove all container content without resetting the DB:
pulpcore-manager shell_plus:

# delete all the repos & distros
ContainerDistribution.objects.all().delete()
ContainerRepository.objects.all().delete()
ContainerPushRepository.objects.all().delete()
# delete all the container content
Tag.objects.all().delete()
ManifestSignature.objects.all().delete()
ManifestListManifest.objects.all().delete()
Manifest.objects.all().delete()
ContainerNamespace.objects.all().delete()
BlobManifest.objects.all().delete()
Blob.objects.all().delete()
# Clean up left over artifacts
Artifact.objects.orphaned(orphan_protection_time=0).delete()

These commands will delete everything container related across all domains.

Ok I did not realize pushing an image create a distribution automatically so I had no cleared up the distribution --domain nautilus distribution linux/lrh9 and had only removed --domain nautilus distribution linux, running an orphan cleanup did then remove the objects albeit I still saw file system in place.

After this I again could push image however, the 500 exits on the pull, I tried a new image as well to confirm there were no issues with still remnants of the older image. I now see the issue is that the content app is not using the correct storage path for the domain to pull the image e.g.

❯ podman push pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9-init:latest
Getting image source signatures
Copying blob 3dbfb22e5a29 done   | 
Copying config 1d409feda6 done   | 
Writing manifest to image destination


❯ podman pull pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9-init:latest
Trying to pull pulp-east.dev.aws.xyz.net/nautilus/linux/lrh9-init:latest...
Error: parsing image configuration: fetching blob: received unexpected HTTP status: 500 Internal Server Error

Looking at the content app we see

  File "/usr/local/lib/python3.12/site-packages/pulp_container/app/registry.py", line 91, in _dispatch
    raise Exception("Expected path '{}' is not found".format(path))
Exception: Expected path '/var/lib/pulp/media/artifact/01965e79-d2b6-707a-950c-367815ac6622/1d/409feda6bbe443a3bf399f91e7399f1fcb361d67858319b073452602d343ed' is not found
100.72.17.160 [02/May/2025:10:18:45 +0000] "GET /pulp/container/nautilus/linux/lrh9-init/blobs/sha256:1d409feda6bbe443a3bf399f91e7399f1fcb361d67858319b073452602d343ed?expires=1746183760&validate_token=8f73482df9380712fa76bf556e2ed1463d93afe1d709967be632e79791148575:6518721029ef732be0a33b9fa5c2d6f3e15ff9e500634e37380115da49a81a84 HTTP/1.1" 500 247 "https://pulp-east.dev.aws.xyz.net/v2/nautilus/linux/lrh9-init/blobs/sha256:1d409feda6bbe443a3bf399f91e7399f1fcb361d67858319b073452602d343ed" "containers/5.32.2 (github.com/containers/image)"

If we look at the path in question

pulp@container[pulp-content-65b55fd74-hwxr8]:/# ls -al /var/lib/pulp/media/artifact/01965e79-d2b6-707a-950c-367815ac6622/1d/409feda6bbe443a3bf399f91e7399f1fcb361d67858319b073452602d343ed
ls: cannot access '/var/lib/pulp/media/artifact/01965e79-d2b6-707a-950c-367815ac6622/1d/409feda6bbe443a3bf399f91e7399f1fcb361d67858319b073452602d343ed': No such file or directory

pulp@container[pulp-content-65b55fd74-hwxr8]:/# ls -al /var/lib/pulp/nautilus/artifact/01965e79-d2b6-707a-950c-367815ac6622/1d/409feda6bbe443a3bf399f91e7399f1fcb361d67858319b073452602d343ed
-rw-rwxr--. 1 pulp pulp 1531 May  2 10:02 /var/lib/pulp/nautilus/artifact/01965e79-d2b6-707a-950c-367815ac6622/1d/409feda6bbe443a3bf399f91e7399f1fcb361d67858319b073452602d343ed

I see from the error it appears to be pulling from the the default domain storage config not the domains storage which push is using, have I goofed some config ?

pulp domain list

- description: Nautilus  Container Repo
  hide_guarded_distributions: false
  name: nautilus
  prn: prn:core.domain:01965e79-d2b6-707a-950c-367815ac6622
  pulp_created: '2025-04-22T17:09:24.278948Z'
  pulp_href: /pulp/default/api/v3/domains/01965e79-d2b6-707a-950c-367815ac6622/
  pulp_labels: {}
  pulp_last_updated: '2025-05-01T15:20:11.092007Z'
  redirect_to_object_storage: false
  storage_class: pulpcore.app.models.storage.FileSystem
  storage_settings:
    base_url: /
    directory_permissions_mode: null
    file_permissions_mode: 444
    hidden_fields: []
    location: /var/lib/pulp/nautilus/
- description: null
  hide_guarded_distributions: false
  name: default
  prn: prn:core.domain:018e141a-42f6-78fb-bc10-5f61ab7a73cb
  pulp_created: '2024-03-06T14:10:49.210591Z'
  pulp_href: /pulp/default/api/v3/domains/018e141a-42f6-78fb-bc10-5f61ab7a73cb/
  pulp_labels: {}
  pulp_last_updated: '2024-03-06T14:10:49.210612Z'
  redirect_to_object_storage: true
  storage_class: pulpcore.app.models.storage.FileSystem
  storage_settings:
    base_url: /
    directory_permissions_mode: null
    file_permissions_mode: 420
    hidden_fields: []
    location: /var/lib/pulp/media

Technically no, you should be able to customize a file storage domain’s location to any folder on your system (with correct permissions), but it seems that our code is expecting that you set it to /var/lib/pulp/media (the default setting). This is probably a bug in our implementation. You can set the location for the nautilus domain to the same folder, Pulp will separate the non-default domain artifacts into their own folder (e.g. /var/lib/pulp/media/artifact/01965e79-d2b6-707a-950c-367815ac6622) based on the domain’s id. I’ll file a bug report for this

1 Like

Here is the issue tracker: https://github.com/pulp/pulp_container/issues/1998

2 Likes

Thanks, that does work around the issue. Thanks for opening the issue will track.

2 Likes