Thanks for the information. At this stage, I cannot really devote time on this development activity, but I could provide you with more debugging information/logs if needed.
Can you try out this fix: Fix sync with tls_validation=False by gerrod3 · Pull Request #656 · pulp/pulp_python · GitHub and see if it works?
1 Like
@Pulp-User-333 I went ahead and release the fix in pulp_python 3.11.1 (pulp-python · PyPI). The fix also includes better logging on sync errors, so try it out and let us know if it works.
1 Like
Thanks. Is the new change also available in the Docker compose deployment as well?
1 Like
If all goes well, It should land in the newest version of the container images within a day.
1 Like
Perfect. Thanks. Once available in the containers, I will test and let you know! The latest one now has: Python:3.11.0. But I will check again later.
I was able to get the latest containerized verion of Pulp today.
pulpcore: 3.50.2
python: 3.11.1
The remote has its proxy set and also:
“tls_validation”: false,
“includes”: [
“django>=4.2”,
“nltk>=3.8”,
“numpy~=1.24.3”,
“pip-tools~=6.13.0”,
“scipy>=1.10.1”,
“torch~=2.0.1”
],
When policy is on_demand:
Task seems to be compelted successfully:
{
“pulp_href”: “/pulp/api/v3/tasks/018ee389-b66b-7227-ab95-c62de5dc9a29/”,
“pulp_created”: “2024-04-15T20:53:56.973257Z”,
“pulp_last_updated”: “2024-04-15T20:53:56.973277Z”,
“state”: “completed”,
“name”: “pulp_python.app.tasks.sync.sync”,
“logging_cid”: “4f6e15bcadc44267921ce2ca4b2be5a0”,
“created_by”: “/pulp/api/v3/users/1/”,
“unblocked_at”: “2024-04-15T20:53:57.010784Z”,
“started_at”: “2024-04-15T20:53:57.088526Z”,
“finished_at”: “2024-04-15T20:54:02.943775Z”,
“error”: null,
“worker”: “/pulp/api/v3/workers/018ee386-974f-744c-b5e4-f9e644f6767c/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [
{
“message”: “Fetching Project Metadata”,
“code”: “sync.fetching.project”,
“state”: “completed”,
“total”: null,
“done”: 6,
“suffix”: null
},
{
“message”: “Downloading Artifacts”,
“code”: “sync.downloading.artifacts”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
},
{
“message”: “Associating Content”,
“code”: “associating.content”,
“state”: “completed”,
“total”: null,
“done”: 300,
“suffix”: null
}
],
“created_resources”: [
“/pulp/api/v3/repositories/python/python/018ee389-869e-70a9-9224-cac4164b5dea/versions/1/”
],
“reserved_resources_record”: [
“prn:python.pythonrepository:018ee389-869e-70a9-9224-cac4164b5dea”,
“/pulp/api/v3/repositories/python/python/018ee389-869e-70a9-9224-cac4164b5dea/”,
“shared:prn:python.pythonremote:018ee389-9aff-7609-916e-bad057e4ad8e”,
“shared:/pulp/api/v3/remotes/python/python/018ee389-9aff-7609-916e-bad057e4ad8e/”,
“shared:prn:core.domain:018ee385-4790-7caf-8e12-ee74839a22ed”,
“shared:/pulp/api/v3/domains/018ee385-4790-7caf-8e12-ee74839a22ed/”
]
}
However, after publishing and distributing the repository, we are getting:
“500 Internal Server Error”
here are the logs:
pulp-pulp_web-1 | 2024-04-15T21:05:39.594330762Z 172.27.0.1 - - [15/Apr/2024:21:05:39 +0000] “GET /pulp/content/PyPI/ HTTP/1.1” 500 145 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36”
pulp-pulp_content-2 | 2024-04-15T21:05:39.593832956Z [2024-04-15 21:05:39 +0000] [23] [ERROR] Error handling request
pulp-pulp_content-2 | 2024-04-15T21:05:39.593885724Z Traceback (most recent call last):
pulp-pulp_content-2 | 2024-04-15T21:05:39.593921657Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 518, in thread_handler
pulp-pulp_content-2 | 2024-04-15T21:05:39.593936583Z raise exc_info[1]
pulp-pulp_content-2 | 2024-04-15T21:05:39.593950331Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 518, in thread_handler
pulp-pulp_content-2 | 2024-04-15T21:05:39.593964038Z raise exc_info[1]
pulp-pulp_content-2 | 2024-04-15T21:05:39.593977328Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 619, in _match_and_stream
pulp-pulp_content-2 | 2024-04-15T21:05:39.593991155Z await publication.published_artifact.aget(relative_path=index_path)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594004615Z File “/usr/local/lib/python3.9/site-packages/django/db/models/query.py”, line 649, in aget
pulp-pulp_content-2 | 2024-04-15T21:05:39.594018292Z return await sync_to_async(self.get)(*args, **kwargs)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594031662Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 468, in call
pulp-pulp_content-2 | 2024-04-15T21:05:39.594045249Z ret = await asyncio.shield(exec_coro)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594058582Z File “/usr/lib64/python3.9/concurrent/futures/thread.py”, line 58, in run
pulp-pulp_content-2 | 2024-04-15T21:05:39.594072089Z result = self.fn(*self.args, **self.kwargs)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594085385Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 522, in thread_handler
pulp-pulp_content-2 | 2024-04-15T21:05:39.594101865Z return func(*args, **kwargs)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594115178Z File “/usr/local/lib/python3.9/site-packages/django/db/models/query.py”, line 637, in get
pulp-pulp_content-2 | 2024-04-15T21:05:39.594128842Z raise self.model.DoesNotExist(
pulp-pulp_content-2 | 2024-04-15T21:05:39.594142028Z pulpcore.app.models.publication.PublishedArtifact.DoesNotExist: PublishedArtifact matching query does not exist.
pulp-pulp_content-2 | 2024-04-15T21:05:39.594155641Z
pulp-pulp_content-2 | 2024-04-15T21:05:39.594168727Z During handling of the above exception, another exception occurred:
pulp-pulp_content-2 | 2024-04-15T21:05:39.594182000Z
pulp-pulp_content-2 | 2024-04-15T21:05:39.594195240Z Traceback (most recent call last):
pulp-pulp_content-2 | 2024-04-15T21:05:39.594208643Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py”, line 452, in _handle_request
pulp-pulp_content-2 | 2024-04-15T21:05:39.594222474Z resp = await request_handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594235734Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py”, line 543, in _handle
pulp-pulp_content-2 | 2024-04-15T21:05:39.594249417Z resp = await handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594262587Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py”, line 114, in impl
pulp-pulp_content-2 | 2024-04-15T21:05:39.594276327Z return await handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594289513Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py”, line 48, in authenticate
pulp-pulp_content-2 | 2024-04-15T21:05:39.594303214Z return await handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594316446Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/instrumentation.py”, line 230, in middleware
pulp-pulp_content-2 | 2024-04-15T21:05:39.594350215Z resp = await handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594364456Z File “/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py”, line 352, in cached_function
pulp-pulp_content-2 | 2024-04-15T21:05:39.594378317Z response = await self.make_entry(
pulp-pulp_content-2 | 2024-04-15T21:05:39.594391784Z File “/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py”, line 395, in make_entry
pulp-pulp_content-2 | 2024-04-15T21:05:39.594405618Z response = await handler(*args, **kwargs)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594418922Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 257, in stream_content
pulp-pulp_content-2 | 2024-04-15T21:05:39.594432605Z return await self._match_and_stream(path, request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594446108Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 636, in _match_and_stream
pulp-pulp_content-2 | 2024-04-15T21:05:39.594459809Z body=self.render_html(
pulp-pulp_content-2 | 2024-04-15T21:05:39.594473018Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 438, in render_html
pulp-pulp_content-2 | 2024-04-15T21:05:39.594486666Z return template.render(
pulp-pulp_content-2 | 2024-04-15T21:05:39.594499875Z File “/usr/local/lib/python3.9/site-packages/jinja2/environment.py”, line 1301, in render
pulp-pulp_content-2 | 2024-04-15T21:05:39.594513643Z self.environment.handle_exception()
pulp-pulp_content-2 | 2024-04-15T21:05:39.594527249Z File “/usr/local/lib/python3.9/site-packages/jinja2/environment.py”, line 936, in handle_exception
pulp-pulp_content-2 | 2024-04-15T21:05:39.594541338Z raise rewrite_traceback_stack(source=source)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594554687Z File “”, line 15, in top-level template code
pulp-pulp_content-2 | 2024-04-15T21:05:39.594568722Z File “/usr/local/lib/python3.9/site-packages/jinja2/filters.py”, line 683, in do_filesizeformat
pulp-pulp_content-2 | 2024-04-15T21:05:39.594582516Z bytes = float(value)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594595729Z ValueError: could not convert string to float: ‘None\n’
pulp-pulp_content-2 | 2024-04-15T21:05:39.594309233Z ::ffff:172.27.0.12 [15/Apr/2024:21:05:39 +0000] “GET /pulp/content/PyPI/ HTTP/1.0” 500 315 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36”
If I change the policy to immediate:
Task fails with the following error:
“error”: {
“traceback”: " File “/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py”, line 70, in _execute_task\n result = func(*args, **kwargs)\n File “/usr/local/lib/python3.9/site-packages/pulp_python/app/tasks/sync.py”, line 61, in sync\n DeclarativeVersion(first_stage, repository, mirror).create()\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 161, in create\n loop.run_until_complete(pipeline)\n File “/usr/lib64/python3.9/asyncio/base_events.py”, line 647, in run_until_complete\n return future.result()\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py”, line 220, in create_pipeline\n await asyncio.gather(*futures)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py”, line 41, in call\n await self.run()\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 186, in run\n pb.done += task.result() # download_count\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 241, in _handle_content_unit\n await asyncio.gather(*downloaders_for_content)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/models.py”, line 119, in download\n raise e\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/models.py”, line 111, in download\n download_result = await downloader.run(extra_data=self.extra_data)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 269, in run\n return await download_wrapper()\n File “/usr/local/lib/python3.9/site-packages/backoff/_async.py”, line 151, in retry\n ret = await target(*args, **kwargs)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 254, in download_wrapper\n return await self._run(extra_data=extra_data)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 290, in _run\n self.raise_for_status(response)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 187, in raise_for_status\n response.raise_for_status()\n File “/usr/local/lib64/python3.9/site-packages/aiohttp/client_reqrep.py”, line 1060, in raise_for_status\n raise ClientResponseError(\n",
“description”: “403, message=‘InternalAVError’, url=URL(‘https://files.pythonhosted.org/packages/21/33/4925decd863ce88ed9190a4bd872b01c146243ee68db08c72923984fe335/torch-2.0.1-cp310-cp310-manylinux2014_aarch64.whl’)”
},
Not sure why we are getting 403 from upstream for some of the packages. I can see that some are being downloaded successfully though.