Problem:
Pulp Python on_demand sync errors out when accessing the distribution via the web ui:
500 Internal Server Error
Server got itself in trouble
Expected outcome:
Content of Pulp Python repository is displayed when accessing the distribution via the web ui.
Pulpcore version:
core 3.53.0
Pulp plugins installed and their versions:
python 3.11.1
Operating system - distribution and version:
AlmaLinux - 8.9
Pulp is installed via docker compose
Other relevant data:
Remote:
{
“pulp_href”: “/pulp/api/v3/remotes/python/python/018f829c-c3c5-7b70-9419-44417205adc9/”,
“pulp_created”: “2024-05-16T18:14:22.918354Z”,
“pulp_last_updated”: “2024-05-16T18:14:22.918375Z”,
“name”: “TEST-PyPI”,
“url”: “https://pypi.org/”,
“ca_cert”: null,
“client_cert”: null,
“tls_validation”: false,
“proxy_url”: “x.x.x.x”,
“pulp_labels”: {},
“download_concurrency”: null,
“max_retries”: null,
“policy”: “on_demand”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: false
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“includes”: [
“django>=4.2”,
“nltk>=3.8”,
“numpy~=1.24.3”,
“pip-tools~=6.13.0”,
“scipy>=1.10.1”,
“torch~=2.0.1”
],
“excludes”: [],
“prereleases”: false,
“package_types”: [
“sdist”,
“bdist_wheel”
],
“keep_latest_packages”: 0,
“exclude_platforms”: []
}
Sync Tasks completes successfully:
{
“pulp_href”: “/pulp/api/v3/tasks/018f829c-fac3-7ab0-914e-bface6337fc4/”,
“pulp_created”: “2024-05-16T18:14:36.996727Z”,
“pulp_last_updated”: “2024-05-16T18:14:36.996755Z”,
“state”: “completed”,
“name”: “pulpcore.app.tasks.base.general_create”,
“logging_cid”: “eb5e3f50c1f540b4934e70181340e725”,
“created_by”: “/pulp/api/v3/users/1/”,
“unblocked_at”: “2024-05-16T18:14:37.022698Z”,
“started_at”: “2024-05-16T18:14:37.111315Z”,
“finished_at”: “2024-05-16T18:14:37.462159Z”,
“error”: null,
“worker”: “/pulp/api/v3/workers/018f7cad-514f-7e0c-b462-2080e062b45a/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [],
“created_resources”: [
“/pulp/api/v3/distributions/python/pypi/018f829c-fc89-761a-8cdf-049ee96b1414/”
],
“reserved_resources_record”: [
“pdrn:018f7cab-fc28-7e6e-ad2c-f8ac40de37e3:distributions”,
“/api/v3/distributions/”,
“shared:prn:core.domain:018f7cab-fc28-7e6e-ad2c-f8ac40de37e3”,
“shared:/pulp/api/v3/domains/018f7cab-fc28-7e6e-ad2c-f8ac40de37e3/”
]
}
Publication and distributions are also created successfully.
Repository version is also increased after the sync:
{
“pulp_href”: “/pulp/api/v3/repositories/python/python/018f829c-afc3-7630-8572-3daf6d886da3/”,
“pulp_created”: “2024-05-16T18:14:17.796363Z”,
“pulp_last_updated”: “2024-05-16T18:14:31.527041Z”,
“versions_href”: “/pulp/api/v3/repositories/python/python/018f829c-afc3-7630-8572-3daf6d886da3/versions/”,
“pulp_labels”: {
“OS”: “PyPI”,
“environment”: “Test”
},
“latest_version_href”: “/pulp/api/v3/repositories/python/python/018f829c-afc3-7630-8572-3daf6d886da3/versions/1/”,
“name”: “TEST-PyPI”,
“description”: “TEST PyPI repository”,
“retain_repo_versions”: null,
“remote”: “/pulp/api/v3/remotes/python/python/018f829c-c3c5-7b70-9419-44417205adc9/”,
“autopublish”: false
}
However, accessing the web UI for the Python repository distribution generates the following error:
500 Internal Server Error
Server got itself in trouble
Logs:
pulp-pulp_content-1 | 2024-05-16T18:20:34.457501476Z [2024-05-16 18:20:34 +0000] [22] [ERROR] Error handling request
pulp-pulp_content-1 | 2024-05-16T18:20:34.457534249Z Traceback (most recent call last):
pulp-pulp_content-1 | 2024-05-16T18:20:34.457541904Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 518, in thread_handler
pulp-pulp_content-1 | 2024-05-16T18:20:34.457548712Z raise exc_info[1]
pulp-pulp_content-1 | 2024-05-16T18:20:34.457554986Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 518, in thread_handler
pulp-pulp_content-1 | 2024-05-16T18:20:34.457561780Z raise exc_info[1]
pulp-pulp_content-1 | 2024-05-16T18:20:34.457568090Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 619, in _match_and_stream
pulp-pulp_content-1 | 2024-05-16T18:20:34.457588677Z await publication.published_artifact.aget(relative_path=index_path)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457595560Z File “/usr/local/lib/python3.9/site-packages/django/db/models/query.py”, line 649, in aget
pulp-pulp_content-1 | 2024-05-16T18:20:34.457601973Z return await sync_to_async(self.get)(*args, **kwargs)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457608218Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 468, in call
pulp-pulp_content-1 | 2024-05-16T18:20:34.457614538Z ret = await asyncio.shield(exec_coro)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457620750Z File “/usr/lib64/python3.9/concurrent/futures/thread.py”, line 58, in run
pulp-pulp_content-1 | 2024-05-16T18:20:34.457627116Z result = self.fn(*self.args, **self.kwargs)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457633366Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 522, in thread_handler
pulp-pulp_content-1 | 2024-05-16T18:20:34.457639741Z return func(*args, *kwargs)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457645955Z File “/usr/local/lib/python3.9/site-packages/django/db/models/query.py”, line 637, in get
pulp-pulp_content-1 | 2024-05-16T18:20:34.457652291Z raise self.model.DoesNotExist(
pulp-pulp_content-1 | 2024-05-16T18:20:34.457658461Z pulpcore.app.models.publication.PublishedArtifact.DoesNotExist: PublishedArtifact matching query does not exist.
pulp-pulp_content-1 | 2024-05-16T18:20:34.457664847Z
pulp-pulp_content-1 | 2024-05-16T18:20:34.457670967Z During handling of the above exception, another exception occurred:
pulp-pulp_content-1 | 2024-05-16T18:20:34.457679524Z
pulp-pulp_content-1 | 2024-05-16T18:20:34.457685597Z Traceback (most recent call last):
pulp-pulp_content-1 | 2024-05-16T18:20:34.457691775Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py”, line 452, in _handle_request
pulp-pulp_content-1 | 2024-05-16T18:20:34.457698148Z resp = await request_handler(request)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457704283Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py”, line 543, in _handle
pulp-pulp_web-1 | 2024-05-16T18:20:34.457981375Z - - [16/May/2024:18:20:34 +0000] “GET /pulp/content/PyPI/TEST/ HTTP/1.1” 500 145 “http://localhost:8080/pulp/content/PyPI/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36”
pulp-pulp_content-1 | 2024-05-16T18:20:34.457710648Z resp = await handler(request)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457716789Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py”, line 114, in impl
pulp-pulp_content-1 | 2024-05-16T18:20:34.457723229Z return await handler(request)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457729838Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py”, line 48, in authenticate
pulp-pulp_content-1 | 2024-05-16T18:20:34.457736229Z return await handler(request)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457742497Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/instrumentation.py”, line 230, in middleware
pulp-pulp_content-1 | 2024-05-16T18:20:34.457749856Z resp = await handler(request)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457756137Z File “/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py”, line 352, in cached_function
pulp-pulp_content-1 | 2024-05-16T18:20:34.457762534Z response = await self.make_entry(
pulp-pulp_content-1 | 2024-05-16T18:20:34.457768676Z File “/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py”, line 395, in make_entry
pulp-pulp_content-1 | 2024-05-16T18:20:34.457775039Z response = await handler(args, kwargs)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457789355Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 257, in stream_content
pulp-pulp_content-1 | 2024-05-16T18:20:34.457796079Z return await self._match_and_stream(path, request)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457802295Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 636, in _match_and_stream
pulp-pulp_content-1 | 2024-05-16T18:20:34.457808812Z body=self.render_html(
pulp-pulp_content-1 | 2024-05-16T18:20:34.457814983Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 438, in render_html
pulp-pulp_content-1 | 2024-05-16T18:20:34.457821380Z return template.render(
pulp-pulp_content-1 | 2024-05-16T18:20:34.457827519Z File “/usr/local/lib/python3.9/site-packages/jinja2/environment.py”, line 1301, in render
pulp-pulp_content-1 | 2024-05-16T18:20:34.457833946Z self.environment.handle_exception()
pulp-pulp_content-1 | 2024-05-16T18:20:34.457840161Z File “/usr/local/lib/python3.9/site-packages/jinja2/environment.py”, line 936, in handle_exception
pulp-pulp_content-1 | 2024-05-16T18:20:34.457846544Z raise rewrite_traceback_stack(source=source)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457852766Z File “”, line 15, in top-level template code
pulp-pulp_content-1 | 2024-05-16T18:20:34.457859523Z File “/usr/local/lib/python3.9/site-packages/jinja2/filters.py”, line 683, in do_filesizeformat
pulp-pulp_content-1 | 2024-05-16T18:20:34.457865872Z bytes = float(value)
pulp-pulp_content-1 | 2024-05-16T18:20:34.457871974Z ValueError: could not convert string to float: ‘None\n’
pulp-pulp_content-1 | 2024-05-16T18:20:34.459566657Z ::ffff:172.21.0.12 [16/May/2024:18:20:34 +0000] “GET /pulp/content/PyPI/TEST/ HTTP/1.0” 500 315 “http://localhost:8080/pulp/content/PyPI/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36”