Getting 500 internal server error for python plugin (pip install)

Support
#1

Problem:
I am trying to create local python-pypi while connecting it remote pypi. I have followed the official documentation: Setup your own PyPI - Pulp Project and even created a simple bash script to create everything. In addition, I have deployed the operator and Pulp resource:

# Pulp resource with Python and Container plugins
apiVersion: repo-manager.pulpproject.org/v1beta2
kind: Pulp
metadata:
  name: pulp
  namespace: pulp-operator
spec:
  admin_password_secret: pulp-admin-password
  api:
    replicas: 1
  content:
    replicas: 1
  worker:
    replicas: 2
  web:
    replicas: 1

Note: I have created port-forwarding for the following services:

kubectl port-forward svc/pulp-api-svc 24817:24817 -n pulp-operator
kubectl port-forward -n pulp-operator svc/pulp-web-svc 24880:24880

http://localhost:24880/pypi/local-pypi/ returns 200 OK:

{
“projects”: 1,
“releases”: 1,
“files”: 2
}

http://localhost:24880/pulp/api/v3/content/python/packages/ return 200 OK:

{
“count”: 2,
“next”: null,
“previous”: null,
“results”: [
{
“pulp_href”: “/pulp/api/v3/content/python/packages/…/”,
“prn”: “prn:python.pythonpackagecontent:…”,
“pulp_created”: “2025-01-06T13:05:34.266473Z”,
“pulp_last_updated”: “2025-01-06T13:05:34.266487Z”,
“artifact”: null,
“filename”: “shelf_reader-0.1-py2-none-any.whl”,
“packagetype”: “bdist_wheel”,
“name”: “shelf-reader”,
…

I am also able to see the python content packages here (sherlf-reader.whl and .tar) and simple/ but when I try to access simple/ I receive 500:
http://localhost:24880/pulp/content/local-pypi/

# Start by creating a new repository named "local-pypi":
pulp python repository create --name local-pypi

# Create a remote that syncs some versions of shelf-reader into your repository.
pulp python remote create \
  --name remote-pypi \
  --url https://pypi.org/ \
  --includes '["shelf-reader"]'

# Link the remote to the repository
pulp python repository update \
  --name local-pypi \
  --remote remote-pypi

# Using the Remote we just created, we kick off a sync task
pulp python repository sync \
  --name local-pypi \
  --remote remote-pypi


# Create a new publication specifying the repository_version.
pulp python publication create --repository local-pypi

# Publications can only be referenced through their pulp_href
PUBLICATION_HREF=$(pulp python publication list | jq -r .[0].pulp_href)


# Distributions are created asynchronously. Create one, and specify the publication that will
# be served at the base path specified.
pulp python distribution create --name local-pypi-distro --base-path local-pypi --publication "$PUBLICATION_HREF"


# This configures the repository to produce new publications when a new version is created
pulp python repository update \
  --name local-pypi \
  --autopublish

# Link the remote to the distribution for pull-through caching
pulp python distribution update \
  --name local-pypi-distro \
  --remote remote-pypi

Expected outcome: pip3 install --index-url=http://localhost:24880/pypi/local-pypi/simple shelf-reader suppose to work and install the package but I receive:

ERROR: Could not find a version that satisfies the requirement shelf-reader (from versions: none)
ERROR: No matching distribution found for shelf-reader

Pulpcore version:
Pulp3 Command Line Interface, Version 0.29.2

Pulp plugins installed and their versions:
INSTALLED PLUGINS: python

Plugin Versions:
common: 0.29.2

Operating system - distribution and version:
Running inside container in harvester cluster
Server Version: v1.27.13+rke2r1

Other relevant data:

#2

Can you post the stack trace from the 500 error? Also, can you post the plugin versions, pulp status output?

#3

Server error:

::ffff:10.52.2.182 [16/Jan/2025:09:04:18 +0000] “GET /pulp/content/local-pypi/ HTTP/1.0” 200 823 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36”
[2025-01-16 09:04:20 +0000] [18] [ERROR] Error handling request
Traceback (most recent call last):
File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py”, line 480, in _handle_request
resp = await request_handler(request)
File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py”, line 569, in _handle
return await handler(request)
File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py”, line 117, in impl
return await handler(request)
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py”, line 48, in authenticate
return await handler(request)
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 268, in stream_content
return await self._match_and_stream(path, request)
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 669, in _match_and_stream
return await self._serve_content_artifact(ca, headers, request)
File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 998, in _serve_content_artifact
content_length = artifact_file.size
File “/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py”, line 74, in size
return self.storage.size(self.name)
File “/usr/local/lib/python3.9/site-packages/django/core/files/storage/filesystem.py”, line 182, in size
return os.path.getsize(self.path(name))
File “/usr/lib64/python3.9/genericpath.py”, line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: ‘/var/lib/pulp/media/artifact/cb/d30a9b2d7d6031c41ad8ac916068bb1335f515dfbd8bbe1941207d008ad36c’
::ffff:10.52.2.182 [16/Jan/2025:09:04:20 +0000] “GET /pulp/content/local-pypi/simple/ HTTP/1.0” 500 317 “http://localhost:24880/pulp/content/local-pypi/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36”

pulp status

{
  "versions": [
    {
      "component": "core",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulpcore.app",
      "domain_compatible": true
    },
    {
      "component": "python",
      "version": "3.12.5",
      "package": "pulp-python",
      "module": "pulp_python.app",
      "domain_compatible": true
    },
    {
      "component": "ansible",
      "version": "0.23.1",
      "package": "pulp-ansible",
      "module": "pulp_ansible.app",
      "domain_compatible": false
    },
    {
      "component": "rpm",
      "version": "3.27.2",
      "package": "pulp-rpm",
      "module": "pulp_rpm.app",
      "domain_compatible": true
    },
    {
      "component": "ostree",
      "version": "2.4.4",
      "package": "pulp-ostree",
      "module": "pulp_ostree.app",
      "domain_compatible": true
    },
    {
      "component": "deb",
      "version": "3.4.0",
      "package": "pulp_deb",
      "module": "pulp_deb.app",
      "domain_compatible": false
    },
    {
      "component": "container",
      "version": "2.22.0",
      "package": "pulp-container",
      "module": "pulp_container.app",
      "domain_compatible": false
    },
    {
      "component": "maven",
      "version": "0.8.1",
      "package": "pulp-maven",
      "module": "pulp_maven.app",
      "domain_compatible": false
    },
    {
      "component": "certguard",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulp_certguard.app",
      "domain_compatible": true
    },
    {
      "component": "file",
      "version": "3.69.0",
      "package": "pulpcore",
      "module": "pulp_file.app",
      "domain_compatible": true
    }
  ],
  "online_workers": [
    {
      "pulp_href": "/pulp/api/v3/workers/01943bb7-f730-7bbf-8364-75b1784ea44f/",
      "prn": "prn:core.worker:01943bb7-f730-7bbf-8364-75b1784ea44f",
      "pulp_created": "2025-01-06T13:04:59.698418Z",
      "pulp_last_updated": "2025-01-06T13:04:59.698437Z",
      "name": "1@pulp-worker-7f9988cd5d-lk7hx",
      "last_heartbeat": "2025-01-16T09:05:51.425622Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      },
      "current_task": null
    },
    {
      "pulp_href": "/pulp/api/v3/workers/01943bb7-efa6-70df-83e0-a312da1f1936/",
      "prn": "prn:core.worker:01943bb7-efa6-70df-83e0-a312da1f1936",
      "pulp_created": "2025-01-06T13:04:57.768544Z",
      "pulp_last_updated": "2025-01-06T13:04:57.768568Z",
      "name": "1@pulp-worker-7f9988cd5d-lpbp9",
      "last_heartbeat": "2025-01-16T09:05:59.338846Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      },
      "current_task": null
    }
  ],
  "online_api_apps": [
    {
      "name": "15@pulp-api-74cd9b5b87-p7wz8",
      "last_heartbeat": "2025-01-16T09:05:59.611872Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    },
    {
      "name": "14@pulp-api-74cd9b5b87-p7wz8",
      "last_heartbeat": "2025-01-16T09:05:59.612589Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    }
  ],
  "online_content_apps": [
    {
      "name": "16@pulp-content-9f8cd6fd8-pvwp7",
      "last_heartbeat": "2025-01-16T09:05:53.245172Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    },
    {
      "name": "18@pulp-content-9f8cd6fd8-pvwp7",
      "last_heartbeat": "2025-01-16T09:05:56.318834Z",
      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": false
  },
  "storage": {
    "total": 1167500771328,
    "used": 58927759360,
    "free": 1049191788544
  },
  "content_settings": {
    "content_origin": "http://pulp-web-svc.pulp-operator.svc.cluster.local:24880",
    "content_path_prefix": "/pulp/content/"
  },
  "domain_enabled": false
}

I have tried to check if the content exists and received the following:

kubectl exec -n pulp-operator deployment/pulp-content -- ls -la /var/lib/pulp/media/artifact/
total 20
drwxr-sr-x 4 pulp root 4096 Jan 13 12:24 .
drwxrwsr-x 1 pulp root 4096 Jan 13 11:32 ..
drwxr-sr-x 2 pulp root 4096 Jan 13 11:32 04
drwxr-sr-x 2 pulp root 4096 Jan 13 12:24 2e
#4

Can you update the remote to immediate (python remotes are on_demand by default) and then resync? I have a feeling your file storage backend on the operator isn’t set up properly. The sync will download the artifacts to your storage and if it fails then we should know that it isn’t set up correctly.

pulp python remote update --name remote-pypi --policy immediate
pulp python repository sync --name local-pypi --remote remote-pypi
#5

I did it and still same issue:
kubectl logs -n pulp-operator pulp-web-56d998d5d-zczzc

127.0.0.1 - - [19/Jan/2025:08:37:53 +0000] "GET /pypi HTTP/1.1" 404 179 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /pypi/local-pypi/ HTTP/1.1" 200 10139 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/css/bootstrap-tweaks.css HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/css/default.css HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/jquery-3.7.1.min.js HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/css/prettify.css HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/ajax-form.js HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/css/bootstrap.min.css HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/csrf.js HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/bootstrap.min.js HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/prettify-min.js HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/default.js HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/load-ajax-form.js HTTP/1.1" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

127.0.0.1 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/img/grid.png HTTP/1.1" 304 0 "http://localhost:24880/assets/rest_framework/css/bootstrap-tweaks.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

172.20.65.102 - - [19/Jan/2025:08:38:00 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3697 "-" "kube-probe/1.27"

127.0.0.1 - - [19/Jan/2025:08:38:06 +0000] "GET /pypi/local-pypi/simple/ HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"

172.20.65.102 - - [19/Jan/2025:08:38:10 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3697 "-" "kube-probe/1.27"

172.20.65.102 - - [19/Jan/2025:08:38:20 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3697 "-" "kube-probe/1.27"

172.20.65.102 - - [19/Jan/2025:08:38:30 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3697 "-" "kube-probe/1.27"

172.20.65.102 - - [19/Jan/2025:08:38:40 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3697 "-" "kube-probe/1.27"

172.20.65.102 - - [19/Jan/2025:08:38:50 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3697 "-" "kube-probe/1.27"

172.20.65.102 - - [19/Jan/2025:08:39:00 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3697 "-" "kube-probe/1.27"

2025/01/19 08:39:07 [info] 8#8: *221876 client timed out (110: Connection timed out) while waiting for request, client: 127.0.0.1, server: 0.0.0.0:8080

kubectl logs -n pulp-operator pulp-api-74cd9b5b87-p7wz8

('pulp [ca7ff2a734584659a4051cf4d0cd4595]: ::ffff:172.20.65.102 - - [19/Jan/2025:08:37:52 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 3697 "-" "kube-probe/1.27"',)

pulp [9a6fb7b5c14b4486825efe182dc496e7]: django.request:WARNING: Not Found: /pypi

('pulp [9a6fb7b5c14b4486825efe182dc496e7]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:53 +0000] "GET /pypi HTTP/1.0" 404 179 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [ba15f29b93d54530b267176741a09be9]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /pypi/local-pypi/ HTTP/1.0" 200 10139 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [e0df938ca1994bedb737d7fdc5dfd100]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/css/bootstrap-tweaks.css HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [bf85d713604248bd9a9cc8efeac301df]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/css/default.css HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [ceecb75ff0474bdbb2e0227518c4cc30]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/jquery-3.7.1.min.js HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [b89b9677a28340a9a1dc2a9ee1a1a81e]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/css/prettify.css HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [166fca58dfe44b6db377861568e6c7c8]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/ajax-form.js HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [3eb44aab3b5d42c2958a10c5de66e6b7]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/css/bootstrap.min.css HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [2a0134cde03d4cb2958e13f183c43332]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/csrf.js HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [064cefb775ab467f991e6f5efb2472f3]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/bootstrap.min.js HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [9432b12c36e64bf9b94478cc6bf31c6c]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/prettify-min.js HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [9745cd5e80064fd8b5987c47735f42d7]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/default.js HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [9b3e66c859eb41cfb41a459cd7cb4d62]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/js/load-ajax-form.js HTTP/1.0" 304 0 "http://localhost:24880/pypi/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [5b99986b05334ca6be0cd2a2598ef1c3]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:37:57 +0000] "GET /assets/rest_framework/img/grid.png HTTP/1.0" 304 0 "http://localhost:24880/assets/rest_framework/css/bootstrap-tweaks.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

('pulp [1847ba320fc24be683ce3ab4e1e8bd21]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:38:00 +0000] "GET /pulp/api/v3/status/ HTTP/1.0" 200 3697 "-" "kube-probe/1.27"',)

('pulp [122604f1db6d4c87846422a44992c5c4]: ::ffff:10.52.2.182 - - [19/Jan/2025:08:38:06 +0000] "GET /pypi/local-pypi/simple/ HTTP/1.0" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"',)

kubectl logs -n pulp-operator pulp-content-9f8cd6fd8-pvwp7

::ffff:10.52.2.182 [19/Jan/2025:08:27:56 +0000] "GET /pulp/content/local-pypi/ HTTP/1.0" 200 823 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
[2025-01-19 08:27:58 +0000] [18] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 480, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py", line 569, in _handle
    return await handler(request)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 268, in stream_content
    return await self._match_and_stream(path, request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 669, in _match_and_stream
    return await self._serve_content_artifact(ca, headers, request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 998, in _serve_content_artifact
    content_length = artifact_file.size
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 74, in size
    return self.storage.size(self.name)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/filesystem.py", line 182, in size
    return os.path.getsize(self.path(name))
  File "/usr/lib64/python3.9/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/cb/d30a9b2d7d6031c41ad8ac916068bb1335f515dfbd8bbe1941207d008ad36c'
::ffff:10.52.2.182 [19/Jan/2025:08:27:58 +0000] "GET /pulp/content/local-pypi/simple/ HTTP/1.0" 500 317 "http://localhost:24880/pulp/content/local-pypi/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='https://pypi.org/flask-app/')
pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='https://pypi.org/flask-app/')
::ffff:10.52.2.182 [19/Jan/2025:08:28:19 +0000] "GET /pulp/content/local-pypi/flask-app/ HTTP/1.0" 404 318 "-" "pip/24.0 {"ci":null,"cpu":"arm64","distro":{"name":"macOS","version":"14.7.2"},"implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL 3.2.1 30 Jan 2024","python":"3.12.2","rustc_version":"1.77.1","system":{"name":"Darwin","release":"23.6.0"}}"
Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='https://pypi.org/shelf-reader/')
pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='https://pypi.org/shelf-reader/')
::ffff:10.52.2.182 [19/Jan/2025:08:30:37 +0000] "GET /pulp/content/local-pypi/shelf-reader/ HTTP/1.0" 404 324 "-" "pip/24.0 {"ci":null,"cpu":"arm64","distro":{"name":"macOS","version":"14.7.2"},"implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL 3.2.1 30 Jan 2024","python":"3.12.2","rustc_version":"1.77.1","system":{"name":"Darwin","release":"23.6.0"}}"
Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='https://pypi.org/shelf-reader/')
pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='https://pypi.org/shelf-reader/')
::ffff:10.52.2.182 [19/Jan/2025:08:31:46 +0000] "GET /pulp/content/local-pypi/shelf-reader/ HTTP/1.0" 404 324 "-" "pip/24.0 {"ci":null,"cpu":"arm64","distro":{"name":"macOS","version":"14.7.2"},"implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL 3.2.1 30 Jan 2024","python":"3.12.2","rustc_version":"1.77.1","system":{"name":"Darwin","release":"23.6.0"}}"
#6

Sorry, could you post the result of the task? pulp task show --uuid RETURNED_TASK_UUID And I would also like to see the new state of the content: pulp python content list --fields 'filename,artifact'

#7

For easier debugging, I have recreated everything and added all the tasks logs:

 bash pypi.sh                   
+ pulp python repository create --name local-pypi
{
  "pulp_href": "/pulp/api/v3/repositories/python/python/01948422-b3d2-7b55-b342-d4f8d8344295/",
  "prn": "prn:python.pythonrepository:01948422-b3d2-7b55-b342-d4f8d8344295",
  "pulp_created": "2025-01-20T14:34:14.356597Z",
  "pulp_last_updated": "2025-01-20T14:34:14.371397Z",
  "versions_href": "/pulp/api/v3/repositories/python/python/01948422-b3d2-7b55-b342-d4f8d8344295/versions/",
  "pulp_labels": {},
  "latest_version_href": "/pulp/api/v3/repositories/python/python/01948422-b3d2-7b55-b342-d4f8d8344295/versions/0/",
  "name": "local-pypi",
  "description": null,
  "retain_repo_versions": null,
  "remote": null,
  "autopublish": false
}
+ pulp python remote create --name remote-pypi --url https://pypi.org/ --includes '["shelf-reader", "django", "flask", "flask-app", "requests", "hvac", "vault"]' --policy immediate
{
  "pulp_href": "/pulp/api/v3/remotes/python/python/01948422-b712-72d7-8caf-e8b7875de617/",
  "prn": "prn:python.pythonremote:01948422-b712-72d7-8caf-e8b7875de617",
  "pulp_created": "2025-01-20T14:34:15.189345Z",
  "pulp_last_updated": "2025-01-20T14:34:15.189396Z",
  "name": "remote-pypi",
  "url": "https://pypi.org/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": null,
  "pulp_labels": {},
  "download_concurrency": null,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
  "hidden_fields": [
    {
      "name": "client_key",
      "is_set": false
    },
    {
      "name": "proxy_username",
      "is_set": false
    },
    {
      "name": "proxy_password",
      "is_set": false
    },
    {
      "name": "username",
      "is_set": false
    },
    {
      "name": "password",
      "is_set": false
    }
  ],
  "includes": [
    "shelf-reader",
    "django",
    "flask",
    "flask-app",
    "requests",
    "hvac",
    "vault"
  ],
  "excludes": [],
  "prereleases": true,
  "package_types": [],
  "keep_latest_packages": 0,
  "exclude_platforms": []
}
+ pulp python repository update --name local-pypi --remote remote-pypi
Started background task /pulp/api/v3/tasks/01948422-c0a6-7019-8c8b-d47b786af6a6/
Done.
+ pulp python repository sync --name local-pypi --remote remote-pypi
Started background task /pulp/api/v3/tasks/01948422-d0b2-7b25-bc67-dc33c03d8cfb/
............................................................................................................................................................................................................................................................................................................Done.
+ pulp python publication create --repository local-pypi
Started background task /pulp/api/v3/tasks/0194842a-ab70-7cc9-b89b-5c7d359f91f0/
.Done.
{
  "pulp_href": "/pulp/api/v3/publications/python/pypi/0194842a-ad22-720d-afd4-14eadee2bfbf/",
  "prn": "prn:python.pythonpublication:0194842a-ad22-720d-afd4-14eadee2bfbf",
  "pulp_created": "2025-01-20T14:42:56.932004Z",
  "pulp_last_updated": "2025-01-20T14:42:57.889983Z",
  "repository_version": "/pulp/api/v3/repositories/python/python/01948422-b3d2-7b55-b342-d4f8d8344295/versions/1/",
  "repository": "/pulp/api/v3/repositories/python/python/01948422-b3d2-7b55-b342-d4f8d8344295/",
  "distributions": []
}
++ pulp python publication list
++ jq -r '.[0].pulp_href'
+ PUBLICATION_HREF=/pulp/api/v3/publications/python/pypi/0194842a-ad22-720d-afd4-14eadee2bfbf/
+ pulp python distribution create --name local-pypi-distro --base-path local-pypi --publication /pulp/api/v3/publications/python/pypi/0194842a-ad22-720d-afd4-14eadee2bfbf/
Started background task /pulp/api/v3/tasks/0194842a-bbab-72e9-b2c5-dbc300cfffca/
.Done.
{
  "pulp_href": "/pulp/api/v3/distributions/python/pypi/0194842a-be1e-7dbf-bc71-024e7a53f531/",
  "prn": "prn:python.pythondistribution:0194842a-be1e-7dbf-bc71-024e7a53f531",
  "pulp_created": "2025-01-20T14:43:01.280237Z",
  "pulp_last_updated": "2025-01-20T14:43:01.280257Z",
  "base_path": "local-pypi",
  "base_url": "https://pulp-api-74cd9b5b87-czpnf/pypi/local-pypi/",
  "content_guard": null,
  "no_content_change_since": "2025-01-20T14:43:01.280257Z",
  "hidden": false,
  "pulp_labels": {},
  "name": "local-pypi-distro",
  "repository": null,
  "publication": "/pulp/api/v3/publications/python/pypi/0194842a-ad22-720d-afd4-14eadee2bfbf/",
  "allow_uploads": true,
  "remote": null
}
+ pulp python repository update --name local-pypi --autopublish
Started background task /pulp/api/v3/tasks/0194842a-cae8-7af8-9640-612948e7c929/
Done.
+ pulp python distribution update --name local-pypi-distro --remote remote-pypi
Started background task /pulp/api/v3/tasks/0194842a-d69e-7e35-9c87-77a00d858ce6/
Done.
(packages) ilyak@ilyak-mac-xm45 Desktop % pulp task list
[
  {
    "pulp_href": "/pulp/api/v3/tasks/0194842a-d69e-7e35-9c87-77a00d858ce6/",
    "prn": "prn:core.task:0194842a-d69e-7e35-9c87-77a00d858ce6",
    "pulp_created": "2025-01-20T14:43:07.551189Z",
    "pulp_last_updated": "2025-01-20T14:43:07.551209Z",
    "state": "completed",
    "name": "pulpcore.app.tasks.base.general_update",
    "logging_cid": "26ff407ad66d42b28fb60ddb0ee9d702",
    "created_by": "/pulp/api/v3/users/1/",
    "unblocked_at": "2025-01-20T14:43:07.723305Z",
    "started_at": "2025-01-20T14:43:07.731101Z",
    "finished_at": "2025-01-20T14:43:07.937118Z",
    "error": null,
    "worker": null,
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [],
    "reserved_resources_record": [
      "pdrn:0194841f-adb7-73af-b814-1e4c69c83896:distributions",
      "shared:prn:core.domain:0194841f-adb7-73af-b814-1e4c69c83896"
    ]
  },
  {
    "pulp_href": "/pulp/api/v3/tasks/0194842a-cae8-7af8-9640-612948e7c929/",
    "prn": "prn:core.task:0194842a-cae8-7af8-9640-612948e7c929",
    "pulp_created": "2025-01-20T14:43:04.553558Z",
    "pulp_last_updated": "2025-01-20T14:43:04.553579Z",
    "state": "completed",
    "name": "pulpcore.app.tasks.base.general_update",
    "logging_cid": "790e46ac67914059afefab132cf8ef13",
    "created_by": "/pulp/api/v3/users/1/",
    "unblocked_at": "2025-01-20T14:43:04.578436Z",
    "started_at": "2025-01-20T14:43:04.588247Z",
    "finished_at": "2025-01-20T14:43:04.760399Z",
    "error": null,
    "worker": null,
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [],
    "reserved_resources_record": [
      "prn:python.pythonrepository:01948422-b3d2-7b55-b342-d4f8d8344295",
      "shared:prn:core.domain:0194841f-adb7-73af-b814-1e4c69c83896"
    ]
  },
  {
    "pulp_href": "/pulp/api/v3/tasks/0194842a-bbab-72e9-b2c5-dbc300cfffca/",
    "prn": "prn:core.task:0194842a-bbab-72e9-b2c5-dbc300cfffca",
    "pulp_created": "2025-01-20T14:43:00.652033Z",
    "pulp_last_updated": "2025-01-20T14:43:00.652051Z",
    "state": "completed",
    "name": "pulpcore.app.tasks.base.general_create",
    "logging_cid": "80564ab31ae24fdea089dff317aa936b",
    "created_by": "/pulp/api/v3/users/1/",
    "unblocked_at": "2025-01-20T14:43:00.832413Z",
    "started_at": "2025-01-20T14:43:00.935249Z",
    "finished_at": "2025-01-20T14:43:01.376576Z",
    "error": null,
    "worker": "/pulp/api/v3/workers/01948422-38a7-7d52-880f-78c7b814c547/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [
      "/pulp/api/v3/distributions/python/pypi/0194842a-be1e-7dbf-bc71-024e7a53f531/"
    ],
    "reserved_resources_record": [
      "pdrn:0194841f-adb7-73af-b814-1e4c69c83896:distributions",
      "shared:prn:core.domain:0194841f-adb7-73af-b814-1e4c69c83896"
    ]
  },
  {
    "pulp_href": "/pulp/api/v3/tasks/0194842a-ab70-7cc9-b89b-5c7d359f91f0/",
    "prn": "prn:core.task:0194842a-ab70-7cc9-b89b-5c7d359f91f0",
    "pulp_created": "2025-01-20T14:42:56.497465Z",
    "pulp_last_updated": "2025-01-20T14:42:56.497527Z",
    "state": "completed",
    "name": "pulp_python.app.tasks.publish.publish",
    "logging_cid": "244aa4b7cf4b40d398ea23fd7d74bd8d",
    "created_by": "/pulp/api/v3/users/1/",
    "unblocked_at": "2025-01-20T14:42:56.711447Z",
    "started_at": "2025-01-20T14:42:56.799826Z",
    "finished_at": "2025-01-20T14:42:57.907669Z",
    "error": null,
    "worker": "/pulp/api/v3/workers/01948422-30eb-7901-9a2e-69bc8966a326/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [
      "/pulp/api/v3/publications/python/pypi/0194842a-ad22-720d-afd4-14eadee2bfbf/"
    ],
    "reserved_resources_record": [
      "shared:prn:python.pythonrepository:01948422-b3d2-7b55-b342-d4f8d8344295",
      "shared:prn:core.domain:0194841f-adb7-73af-b814-1e4c69c83896"
    ]
  },
  {
    "pulp_href": "/pulp/api/v3/tasks/01948422-d0b2-7b25-bc67-dc33c03d8cfb/",
    "prn": "prn:core.task:01948422-d0b2-7b25-bc67-dc33c03d8cfb",
    "pulp_created": "2025-01-20T14:34:21.747010Z",
    "pulp_last_updated": "2025-01-20T14:34:21.747032Z",
    "state": "completed",
    "name": "pulp_python.app.tasks.sync.sync",
    "logging_cid": "04cd3b30f0fb4ae9951f8568e2a28a66",
    "created_by": "/pulp/api/v3/users/1/",
    "unblocked_at": "2025-01-20T14:34:21.947087Z",
    "started_at": "2025-01-20T14:34:22.066788Z",
    "finished_at": "2025-01-20T14:42:54.301025Z",
    "error": null,
    "worker": "/pulp/api/v3/workers/01948422-38a7-7d52-880f-78c7b814c547/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [
      {
        "message": "Downloading Artifacts",
        "code": "sync.downloading.artifacts",
        "state": "completed",
        "total": null,
        "done": 1309,
        "suffix": null
      },
      {
        "message": "Associating Content",
        "code": "associating.content",
        "state": "completed",
        "total": null,
        "done": 1309,
        "suffix": null
      },
      {
        "message": "Fetching Project Metadata",
        "code": "sync.fetching.project",
        "state": "completed",
        "total": null,
        "done": 7,
        "suffix": null
      }
    ],
    "created_resources": [
      "/pulp/api/v3/repositories/python/python/01948422-b3d2-7b55-b342-d4f8d8344295/versions/1/"
    ],
    "reserved_resources_record": [
      "prn:python.pythonrepository:01948422-b3d2-7b55-b342-d4f8d8344295",
      "shared:prn:python.pythonremote:01948422-b712-72d7-8caf-e8b7875de617",
      "shared:prn:core.domain:0194841f-adb7-73af-b814-1e4c69c83896"
    ]
  },
  {
    "pulp_href": "/pulp/api/v3/tasks/01948422-c0a6-7019-8c8b-d47b786af6a6/",
    "prn": "prn:core.task:01948422-c0a6-7019-8c8b-d47b786af6a6",
    "pulp_created": "2025-01-20T14:34:17.639974Z",
    "pulp_last_updated": "2025-01-20T14:34:17.639991Z",
    "state": "completed",
    "name": "pulpcore.app.tasks.base.general_update",
    "logging_cid": "236e9627b28f4b598f9d1b5978836d6a",
    "created_by": "/pulp/api/v3/users/1/",
    "unblocked_at": "2025-01-20T14:34:17.677234Z",
    "started_at": "2025-01-20T14:34:17.702687Z",
    "finished_at": "2025-01-20T14:34:17.880272Z",
    "error": null,
    "worker": null,
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [],
    "reserved_resources_record": [
      "prn:python.pythonrepository:01948422-b3d2-7b55-b342-d4f8d8344295",
      "shared:prn:core.domain:0194841f-adb7-73af-b814-1e4c69c83896"
    ]
  },
  {
    "pulp_href": "/pulp/api/v3/tasks/01948422-58d7-70d4-964f-3bbe4aa40f8e/",
    "prn": "prn:core.task:01948422-58d7-70d4-964f-3bbe4aa40f8e",
    "pulp_created": "2025-01-20T14:33:51.064516Z",
    "pulp_last_updated": "2025-01-20T14:33:51.064530Z",
    "state": "completed",
    "name": "pulpcore.app.tasks.analytics.post_analytics",
    "logging_cid": "09cea94a396a4236991467a743a716bd",
    "created_by": null,
    "unblocked_at": "2025-01-20T14:33:51.099436Z",
    "started_at": "2025-01-20T14:33:51.341761Z",
    "finished_at": "2025-01-20T14:33:52.472780Z",
    "error": null,
    "worker": "/pulp/api/v3/workers/01948422-30eb-7901-9a2e-69bc8966a326/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [],
    "reserved_resources_record": [
      "shared:prn:core.domain:0194841f-adb7-73af-b814-1e4c69c83896"
    ]
  }
]
(packages) ilyak@ilyak-mac-xm45 Desktop % pulp python content list --field 'filename,artifact' 
Not all 1309 entries were shown.
[
  {
    "artifact": "/pulp/api/v3/artifacts/01948429-d892-7ea1-a602-1a839e11a047/",
    "filename": "Django-4.1.5.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-5869-7499-bbf0-83250d31f1ce/",
    "filename": "Django-5.0.6.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948429-ab81-7699-82b1-e0db5c094665/",
    "filename": "Django-4.0rc1.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-2ec9-7d61-9e90-5e8a198cdf69/",
    "filename": "Django-4.2rc1-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948428-3cd9-703e-83a3-c880001b6bc7/",
    "filename": "Django-3.0.12.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-0a29-7d2a-82cb-1b4b4c41fb47/",
    "filename": "Django-4.2.14.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-61aa-7eaa-a7a4-851254e2bf51/",
    "filename": "Django-5.0b1-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948427-de8d-799f-9e5d-bdd0f41af2af/",
    "filename": "Django-2.2.8-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948428-d231-78a4-984a-7b83eca821d1/",
    "filename": "Django-3.1.8.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-1312-75d8-87c1-39d3964b748b/",
    "filename": "Django-4.2.3-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948429-8fb2-76f3-b59e-8f513075b7a8/",
    "filename": "Django-4.0.5.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948428-ffbb-78b5-a492-4fefb2fc9f88/",
    "filename": "Django-3.2.15.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948429-5f83-7b6d-80f0-95dd37cdbee8/",
    "filename": "Django-3.2.8.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948429-7046-7c44-a0c9-f78e64ffcbb4/",
    "filename": "Django-3.2a1.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948428-bf30-7cd0-a8db-4078fe0b1538/",
    "filename": "Django-3.1.9-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-3bdd-7da7-9fcf-bb7ab3990f4e/",
    "filename": "Django-5.0.1-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948429-d9a9-7aa3-a4c0-18f6b89deaf8/",
    "filename": "Django-4.1.7-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948429-623f-725f-84a1-52fae5fd4410/",
    "filename": "Django-3.2.6.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-5e64-722b-a596-5dc9f62f49bd/",
    "filename": "Django-5.0.9-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-569a-75e1-b46d-75cf3d341bec/",
    "filename": "Django-5.0.7-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-754b-7540-8ab4-57ef0694603c/",
    "filename": "Django-5.1.2-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948428-bd44-702d-bc62-dfaa9436ea0f/",
    "filename": "Django-3.1.5-py3-none-any.whl"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948428-a9c8-74f5-88cf-9e0c0c065d08/",
    "filename": "Django-3.1.4.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/0194842a-8a2b-7656-b0cd-2488c1d20648/",
    "filename": "Django-5.2a1.tar.gz"
  },
  {
    "artifact": "/pulp/api/v3/artifacts/01948429-efa8-744d-9f94-b9e147dce947/",
    "filename": "Django-4.1b1.tar.gz"
  }
]

When I try to install the library directly from the content lib it works, but it seems that there’s problem with the publication & distribution (simple indexing)

pip3 install http://pulp-web-svc.pulp-operator.svc.cluster.local:24880/pulp/content/local-pypi/vault-1.2.10-py3-none-any.whl 
Downloading ...

But when I am trying to install with pip3 -i and package name, I receive:

pip3 install --index-url http://localhost:24880/pulp/content/local-pypi/ vault --trusted-host=localhost

Looking in indexes: http://localhost:24880/pulp/content/local-pypi/

ERROR: Could not find a version that satisfies the requirement vault (from versions: none)

ERROR: No matching distribution found for vault

[notice] A new release of pip is available: 24.0 -> 24.3.1

[notice] To update, run: pip install --upgrade pip

pip3 install --index-url http://localhost:24880/pulp/content/local-pypi/simple vault --trusted-host=localhost

Looking in indexes: http://localhost:24880/pulp/content/local-pypi/simple

ERROR: Could not find a version that satisfies the requirement vault (from versions: none)

ERROR: No matching distribution found for vault

[notice] A new release of pip is available: 24.0 -> 24.3.1

[notice] To update, run: pip install --upgrade pip

pip3 install --index-url http://localhost:24880/pypi/local-pypi/simple vault --trusted-host=localhost

Looking in indexes: http://localhost:24880/pypi/local-pypi/simple

ERROR: Could not find a version that satisfies the requirement vault (from versions: none)

ERROR: No matching distribution found for vault

[notice] A new release of pip is available: 24.0 -> 24.3.1

[notice] To update, run: pip install --upgrade pip
#8

If you curl http://pulp-web-svc.pulp-operator.svc.cluster.local:24880/pulp/content/local-pypi/vault-1.2.10-py3-none-any.whl do you still get a 500 error or is it 200 now?

How about curl http://pulp-web-svc.pulp-operator.svc.cluster.local:24880/pulp/content/local-pypi/simple/? Does this give an error? This is what pip is trying to download to find the available packages.

#9

If you curl http://pulp-web-svc.pulp-operator.svc.cluster.local:24880/pulp/content/local-pypi/vault-1.2.10-py3-none-any.whl do you still get a 500 error or is it 200 now?

How about curl http://pulp-web-svc.pulp-operator.svc.cluster.local:24880/pulp/content/local-pypi/simple/? Does this give an error? This is what pip is trying to download to find the available packages.

Both of them return 500

#10

Update: Running the same script using single container (docker) works, looks like the problem seems to be in multi-tenancy (inside k8s cluster)

#11

Checking the Pulp resource provided, it seems like there is no definition of storage for Pulp pods. If that is the case, this is an issue with emptyDir volumes not being accessible between the pods (the files in /var/lib/pulp from pulp-api pod A are not shared with pulp-content pod B).
This can be “fixed” by configuring Pulp with a Persistent Volume (through a k8s StorageClass or PersistentVolumeClaim) or Object Storage (Azure Blob or Amazon S3).

4 Likes
#12

Checking the Pulp resource provided, it seems like there is no definition of storage for Pulp pods. If that is the case, this is an issue with emptyDir volumes not being accessible between the pods (the files in /var/lib/pulp from pulp-api pod A are not shared with pulp-content pod B).
This can be “fixed” by configuring Pulp with a Persistent Volume (through a k8s StorageClass or PersistentVolumeClaim) or Object Storage (Azure Blob or Amazon S3).

Thanks, it works! :slight_smile:

3 Likes
#13

I had the same issue after switching from “local” storage to S3. Pulp has files in the database, but not in a storage. Fix was cd pulp_storage/media/artifact; aws s3 sync . s3://<bucket>/artifact/

1 Like