Can you paste the task output for the sync?
(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)
Looking at the Fetching Project Metadata
report we see it processed 0 packages despite the remote having two listed in the includes
. This suggests it couldn’t reach PyPI. Can you paste your logs here for this sync? There should be messages detailing what HTTP error message it ran into.
We have updated remote.includes to have only one package listed:
“includes”: [
“shelf-reader”
]
pulp python repository sync --name Test-PyPI --remote Test-PyPI
Logs:
pulp-pulp_web-1 | 2024-03-16T03:07:52.364468307Z 192.168.64.1 - admin [16/Mar/2024:03:07:52 +0000] “GET /pulp/api/v3/repositories/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.1” 200 564 “-” "Pulp-CLI/0.24.0"pulp-pulp_api-2 | 2024-03-16T03:07:52.365079753Z (‘pulp [b46b2674aaec47c4a1c02fd71890c6aa]: ::ffff:192.168.64.12 - admin [16/Mar/2024:03:07:52 +0000] “GET /pulp/api/v3/repositories/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.0” 200 564 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-16T03:07:52.907162151Z 192.168.64.1 - admin [16/Mar/2024:03:07:52 +0000] “GET /pulp/api/v3/remotes/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.1” 200 921 “-” “Pulp-CLI/0.24.0”
pulp-pulp_api-1 | 2024-03-16T03:07:52.907699904Z (‘pulp [b46b2674aaec47c4a1c02fd71890c6aa]: ::ffff:192.168.64.12 - admin [16/Mar/2024:03:07:52 +0000] “GET /pulp/api/v3/remotes/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.0” 200 921 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_api-2 | 2024-03-16T03:07:53.467830868Z (‘pulp [b46b2674aaec47c4a1c02fd71890c6aa]: ::ffff:192.168.64.12 - admin [16/Mar/2024:03:07:53 +0000] “POST /pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/sync/ HTTP/1.0” 202 67 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-16T03:07:53.467953830Z 192.168.64.1 - admin [16/Mar/2024:03:07:53 +0000] “POST /pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/sync/ HTTP/1.1” 202 67 “-” “Pulp-CLI/0.24.0”
pulp-pulp_worker-4 | 2024-03-16T03:07:53.547767927Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: pulpcore.tasking.tasks:INFO: Starting task 018e453a-eceb-7f99-89fd-a3ca6789e1d8
pulp-pulp_worker-4 | 2024-03-16T03:07:53.630862591Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized project plugin allowlist_project, filtering [‘shelf-reader’]
pulp-pulp_worker-4 | 2024-03-16T03:07:53.645181546Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized release plugin allowlist_release, filtering [<Requirement(‘shelf-reader’)>]
pulp-pulp_worker-4 | 2024-03-16T03:07:53.646854518Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized release plugin blocklist_release, filtering []
pulp-pulp_worker-4 | 2024-03-16T03:07:53.652540995Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized prerelease plugin with [re.compile(’.+rc\d+$’), re.compile(’.+a(lpha)?\d+$’), re.compile(’.+b(eta)?\d+$’), re.compile(’.+dev\d+$’)]
pulp-pulp_worker-4 | 2024-03-16T03:07:53.652572418Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized prerelease plugin prerelease_release, filtering all packages
pulp-pulp_worker-4 | 2024-03-16T03:07:53.661182674Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch.mirror:INFO: Syncing with <>
pulp-pulp_worker-4 | 2024-03-16T03:07:53.661248245Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch.mirror:INFO: No metadata filters are enabled. Skipping metadata filtering
pulp-pulp_worker-4 | 2024-03-16T03:07:53.661403643Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch.mirror:INFO: No release file filters are enabled. Skipping release file filtering
pulp-pulp_worker-4 | 2024-03-16T03:07:53.661863283Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch.package:INFO: Fetching metadata for package: shelf-reader (serial 0)
pulp-pulp_worker-4 | 2024-03-16T03:07:53.664973114Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: aiohttp.client:WARNING: Could not read .netrc file: [Errno 2] No such file or directory: ‘.fake-netrc’
pulp-pulp_worker-4 | 2024-03-16T03:07:53.665557572Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: aiohttp.client:WARNING: Could not read .netrc file: [Errno 2] No such file or directory: ‘.fake-netrc’
pulp-pulp_worker-4 | 2024-03-16T03:07:53.860220482Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: pulpcore.tasking.tasks:INFO: Task completed 018e453a-eceb-7f99-89fd-a3ca6789e1d8
pulp-pulp_web-1 | 2024-03-16T03:07:54.045535571Z 192.168.64.1 - admin [16/Mar/2024:03:07:54 +0000] “GET /pulp/api/v3/tasks/018e453a-eceb-7f99-89fd-a3ca6789e1d8/ HTTP/1.1” 200 1166 “-” “Pulp-CLI/0.24.0”
pulp-pulp_api-1 | 2024-03-16T03:07:54.046062910Z (‘pulp [b46b2674aaec47c4a1c02fd71890c6aa]: ::ffff:192.168.64.12 - admin [16/Mar/2024:03:07:54 +0000] “GET /pulp/api/v3/tasks/018e453a-eceb-7f99-89fd-a3ca6789e1d8/ HTTP/1.0” 200 1166 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-16T03:07:54.065133558Z 2024/03/16 03:07:54 [info] 29#0: *20633 client 192.168.64.1 closed keepalive connection
=============================================
The error: [Errno 2] No such file or directory: ‘.fake-netrc’ above is, I believe, intentional implementation in Pulp:
"
# Prevent bandersnatch from reading actual .netrc file, set to nonexistent file
environ[“NETRC”] = f"{path.curdir}/.fake-netrc"
"
Also, as pasted in the previous reply, the task does not report any error either:
pulp task show --uuid 018e453a-eceb-7f99-89fd-a3ca6789e1d8
{
“pulp_href”: “/pulp/api/v3/tasks/018e453a-eceb-7f99-89fd-a3ca6789e1d8/”,
“pulp_created”: “2024-03-16T03:07:53.452019Z”,
“state”: “completed”,
“name”: “pulp_python.app.tasks.sync.sync”,
“logging_cid”: “b46b2674aaec47c4a1c02fd71890c6aa”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: “2024-03-16T03:07:53.542987Z”,
“finished_at”: “2024-03-16T03:07:53.858502Z”,
"error": null,
“worker”: “/pulp/api/v3/workers/018e3f22-d719-77d7-a538-c53dd68c0cd8/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [
{
“message”: “Fetching Project Metadata”,
“code”: “sync.fetching.project”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
},
{
“message”: “Downloading Artifacts”,
“code”: “sync.downloading.artifacts”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
},
{
“message”: “Associating Content”,
“code”: “associating.content”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
}
],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/”,
“shared:/pulp/api/v3/remotes/python/python/018e3f85-b903-783a-a05a-f86fdabce839/”,
“shared:/pulp/api/v3/domains/018e3f21-814e-7d37-ad03-074f05743a5f/”
]
}
Thank you
Sadly those log messages were not that helpful, probably because we currently consume the error and do nothing with it: https://github.com/pulp/pulp_python/blob/main/pulp_python/app/tasks/sync.py#L255-L260. If you can edit these lines and add a logging statement here, we will get a better understanding of what the error is.
Another thing after doing some testing myself, the proxy-url format is specific. aiohttp expects it to be in the format scheme://host:port
, e.g. http://127.0.0.1:8899
. In your proxy’s logs you should see one request to PyPI per package listed in includes.
I made sure the remote proxy url includes the scheme (http).
pulp python remote list
[
{
“pulp_href”: “/pulp/api/v3/remotes/python/python/018e3f85-b903-783a-a05a-f86fdabce839/”,
“pulp_created”: “2024-03-15T00:31:52.068466Z”,
“name”: “Test-PyPI”,
"url": “https://pypi.org/”,
“ca_cert”: null,
“client_cert”: null,
"tls_validation": false,
"proxy_url": “http://10.1.1.9:8080”,
“pulp_labels”: {},
“pulp_last_updated”: “2024-03-18T15:58:31.957989Z”,
“download_concurrency”: null,
“max_retries”: null,
“policy”: “on_demand”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: false
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“includes”: [
“shelf-reader”
],
“excludes”: [],
“prereleases”: true,
“package_types”: [],
“keep_latest_packages”: 0,
“exclude_platforms”: []
}
]
Here are the sync logs when I added the logging of the exception:
pulp python repository sync --name Test-PyPI --remote Test-PyPI
pulp-pulp_web-1 | 2024-03-18T21:47:10.652786064Z 192.168.176.1 - admin [18/Mar/2024:21:47:10 +0000] “GET /pulp/api/v3/repositories/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.1” 200 564 “-” “Pulp-CLI/0.24.0”
pulp-pulp_api-1 | 2024-03-18T21:47:10.653551158Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:10 +0000] “GET /pulp/api/v3/repositories/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.0” 200 564 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:47:11.746604115Z 192.168.176.1 - admin [18/Mar/2024:21:47:11 +0000] “GET /pulp/api/v3/remotes/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.1” 200 927 “-” “Pulp-CLI/0.24.0”
pulp-pulp_api-1 | 2024-03-18T21:47:11.747284862Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:11 +0000] “GET /pulp/api/v3/remotes/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.0” 200 927 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_api-1 | 2024-03-18T21:47:12.324803857Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:12 +0000] “POST /pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/sync/ HTTP/1.0” 202 67 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:47:12.324410264Z 192.168.176.1 - admin [18/Mar/2024:21:47:12 +0000] “POST /pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/sync/ HTTP/1.1” 202 67 “-” “Pulp-CLI/0.24.0”
pulp-pulp_worker-1 | 2024-03-18T21:47:12.418490029Z pulp [3603bba994724b058c5d25aa67d47796]: pulpcore.tasking.tasks:INFO: Starting task 018e5388-682c-7f27-9c78-4732adb78743
pulp-pulp_worker-1 | 2024-03-18T21:47:12.498437946Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch:INFO: Initialized project plugin allowlist_project, filtering [‘shelf-reader’]
pulp-pulp_worker-1 | 2024-03-18T21:47:12.514885930Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch:INFO: Initialized release plugin allowlist_release, filtering [<Requirement(‘shelf-reader’)>]
pulp-pulp_worker-1 | 2024-03-18T21:47:12.516635901Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch:INFO: Initialized release plugin blocklist_release, filtering []
pulp-pulp_worker-1 | 2024-03-18T21:47:12.529957467Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch.mirror:INFO: Syncing with https:[slash][slash]pypi[dot]org.
pulp-pulp_worker-1 | 2024-03-18T21:47:12.529986932Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch.mirror:INFO: No metadata filters are enabled. Skipping metadata filtering
pulp-pulp_worker-1 | 2024-03-18T21:47:12.530143699Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch.mirror:INFO: No release file filters are enabled. Skipping release file filtering
pulp-pulp_worker-1 | 2024-03-18T21:47:12.530519436Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch.package:INFO: Fetching metadata for package: shelf-reader (serial 0)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.533371593Z pulp [3603bba994724b058c5d25aa67d47796]: aiohttp.client:WARNING: Could not read .netrc file: [Errno 2] No such file or directory: ‘.fake-netrc’
pulp-pulp_worker-1 | 2024-03-18T21:47:12.533914395Z pulp [3603bba994724b058c5d25aa67d47796]: aiohttp.client:WARNING: Could not read .netrc file: [Errno 2] No such file or directory: ‘.fake-netrc’
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763058486Z pulp [3603bba994724b058c5d25aa67d47796]: pulp_python.app.tasks.sync:ERROR: Cannot connect to host pypi[dot]org[colon]443 ssl:True [SSLCertVerificationError: (1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)’)]
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763096450Z Traceback (most recent call last):
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763111835Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 1104, in _start_tls_connection
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763126207Z tls_transport = await self._loop.start_tls(
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763139747Z File “/usr/lib64/python3.9/asyncio/base_events.py”, line 1240, in start_tls
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763153561Z await waiter
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763167128Z File “/usr/lib64/python3.9/asyncio/sslproto.py”, line 534, in data_received
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763180892Z ssldata, appdata = self._sslpipe.feed_ssldata(data)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763221799Z File “/usr/lib64/python3.9/asyncio/sslproto.py”, line 188, in feed_ssldata
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763236276Z self._sslobj.do_handshake()
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763249648Z File “/usr/lib64/python3.9/ssl.py”, line 945, in do_handshake
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763263279Z self._sslobj.do_handshake()
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763276656Z ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763290399Z
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763303602Z The above exception was the direct cause of the following exception:
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763317029Z
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763330008Z Traceback (most recent call last):
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763343360Z File “/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py”, line 129, in package_syncer
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763357181Z await package.update_metadata(self.master, attempts=3)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763370651Z File “/usr/local/lib/python3.9/site-packages/bandersnatch/package.py”, line 61, in update_metadata
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763384288Z self._metadata = await master.get_package_metadata(
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763397608Z File “/usr/local/lib/python3.9/site-packages/bandersnatch/master.py”, line 220, in get_package_metadata
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763411422Z metadata_response = await metadata_generator.asend(None)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763424789Z File “/usr/local/lib/python3.9/site-packages/bandersnatch/master.py”, line 132, in get
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763438743Z async with self.session.get(path, **kw) as r:
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763452140Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/client.py”, line 1194, in aenter
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763465773Z self._resp = await self._coro
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763478966Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/client.py”, line 578, in _request
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763492563Z conn = await self._connector.connect(
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763507306Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 544, in connect
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763521066Z proto = await self._create_connection(req, traces, timeout)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763534306Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 909, in _create_connection
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763547943Z _, proto = await self._create_proxy_connection(req, traces, timeout)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763561256Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 1357, in _create_proxy_connection
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763574933Z return await self._start_tls_connection(
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763588173Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 1118, in _start_tls_connection
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763601896Z raise ClientConnectorCertificateError(req.connection_key, exc) from exc
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763642774Z aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host pypi[dot]org[colon]443 ssl:True [SSLCertVerificationError: (1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)’)]
pulp-pulp_api-1 | 2024-03-18T21:47:12.887955498Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:12 +0000] “GET /pulp/api/v3/tasks/018e5388-682c-7f27-9c78-4732adb78743/ HTTP/1.0” 200 1139 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:47:12.888969825Z 192.168.176.1 - admin [18/Mar/2024:21:47:12 +0000] “GET /pulp/api/v3/tasks/018e5388-682c-7f27-9c78-4732adb78743/ HTTP/1.1” 200 1139 “-” “Pulp-CLI/0.24.0”
pulp-pulp_worker-1 | 2024-03-18T21:47:12.964190192Z pulp [3603bba994724b058c5d25aa67d47796]: pulpcore.tasking.tasks:INFO: Task completed 018e5388-682c-7f27-9c78-4732adb78743
pulp-pulp_api-1 | 2024-03-18T21:47:14.461346967Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:14 +0000] “GET /pulp/api/v3/tasks/018e5388-682c-7f27-9c78-4732adb78743/ HTTP/1.0” 200 1166 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:47:14.462688653Z 192.168.176.1 - admin [18/Mar/2024:21:47:14 +0000] “GET /pulp/api/v3/tasks/018e5388-682c-7f27-9c78-4732adb78743/ HTTP/1.1” 200 1166 “-” “Pulp-CLI/0.24.0”
pulp-pulp_web-1 | 2024-03-18T21:47:14.478233165Z 2024/03/18 21:47:14 [info] 29#0: *1 client 192.168.176.1 closed keepalive connection
^R
remitepulp-pulp_api-1 | 2024-03-18T21:49:30.642575292Z (‘pulp [2a6037aedb5f4a6fa186ad1b50bff8be]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:49:30 +0000] “GET /pulp/api/v3/remotes/python/python/?offset=0&limit=25 HTTP/1.0” 200 1913 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:49:30.642580167Z 192.168.176.1 - admin [18/Mar/2024:21:49:30 +0000] “GET /pulp/api/v3/remotes/python/python/?offset=0&limit=25 HTTP/1.1” 200 1913 “-” “Pulp-CLI/0.24.0”
pulp-pulp_web-1 | 2024-03-18T21:49:30.665551733Z 2024/03/18 21:49:30 [info] 29#0: *8 client 192.168.176.1 closed keepalive connection
From the logs above, it seems like it is still trying to perform TLS verification, despite having it set to False on the remote.
Yes I think you are correct, this is probably a bug in pulp_python. We are not telling bandersnatch (the library we use to get the PyPI metadata) to not perform TLS when it has been disabled on the remote. Can you file an issue for this? https://github.com/pulp/pulp_python/issues/new/choose
Done. Thanks.
Pleas let us know if you need assistance with this bug resolution.
If you want to fix the issue, you would do it in this method here[0].
The instructions for setting up a development environment are here[1].
[0] https://github.com/pulp/pulp_python/blob/main/pulp_python/app/tasks/sync.py#L62
[1] Contributing with code - Pulp Project
Thanks for the information. At this stage, I cannot really devote time on this development activity, but I could provide you with more debugging information/logs if needed.
Can you try out this fix: Fix sync with tls_validation=False by gerrod3 · Pull Request #656 · pulp/pulp_python · GitHub and see if it works?
@Pulp-User-333 I went ahead and release the fix in pulp_python 3.11.1 (pulp-python · PyPI). The fix also includes better logging on sync errors, so try it out and let us know if it works.
Thanks. Is the new change also available in the Docker compose deployment as well?
If all goes well, It should land in the newest version of the container images within a day.
Perfect. Thanks. Once available in the containers, I will test and let you know! The latest one now has: Python:3.11.0. But I will check again later.
I was able to get the latest containerized verion of Pulp today.
pulpcore: 3.50.2
python: 3.11.1
The remote has its proxy set and also:
“tls_validation”: false,
“includes”: [
“django>=4.2”,
“nltk>=3.8”,
“numpy~=1.24.3”,
“pip-tools~=6.13.0”,
“scipy>=1.10.1”,
“torch~=2.0.1”
],
When policy is on_demand:
Task seems to be compelted successfully:
{
“pulp_href”: “/pulp/api/v3/tasks/018ee389-b66b-7227-ab95-c62de5dc9a29/”,
“pulp_created”: “2024-04-15T20:53:56.973257Z”,
“pulp_last_updated”: “2024-04-15T20:53:56.973277Z”,
“state”: “completed”,
“name”: “pulp_python.app.tasks.sync.sync”,
“logging_cid”: “4f6e15bcadc44267921ce2ca4b2be5a0”,
“created_by”: “/pulp/api/v3/users/1/”,
“unblocked_at”: “2024-04-15T20:53:57.010784Z”,
“started_at”: “2024-04-15T20:53:57.088526Z”,
“finished_at”: “2024-04-15T20:54:02.943775Z”,
“error”: null,
“worker”: “/pulp/api/v3/workers/018ee386-974f-744c-b5e4-f9e644f6767c/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [
{
“message”: “Fetching Project Metadata”,
“code”: “sync.fetching.project”,
“state”: “completed”,
“total”: null,
“done”: 6,
“suffix”: null
},
{
“message”: “Downloading Artifacts”,
“code”: “sync.downloading.artifacts”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
},
{
“message”: “Associating Content”,
“code”: “associating.content”,
“state”: “completed”,
“total”: null,
“done”: 300,
“suffix”: null
}
],
“created_resources”: [
“/pulp/api/v3/repositories/python/python/018ee389-869e-70a9-9224-cac4164b5dea/versions/1/”
],
“reserved_resources_record”: [
“prn:python.pythonrepository:018ee389-869e-70a9-9224-cac4164b5dea”,
“/pulp/api/v3/repositories/python/python/018ee389-869e-70a9-9224-cac4164b5dea/”,
“shared:prn:python.pythonremote:018ee389-9aff-7609-916e-bad057e4ad8e”,
“shared:/pulp/api/v3/remotes/python/python/018ee389-9aff-7609-916e-bad057e4ad8e/”,
“shared:prn:core.domain:018ee385-4790-7caf-8e12-ee74839a22ed”,
“shared:/pulp/api/v3/domains/018ee385-4790-7caf-8e12-ee74839a22ed/”
]
}
However, after publishing and distributing the repository, we are getting:
“500 Internal Server Error”
here are the logs:
pulp-pulp_web-1 | 2024-04-15T21:05:39.594330762Z 172.27.0.1 - - [15/Apr/2024:21:05:39 +0000] “GET /pulp/content/PyPI/ HTTP/1.1” 500 145 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36”
pulp-pulp_content-2 | 2024-04-15T21:05:39.593832956Z [2024-04-15 21:05:39 +0000] [23] [ERROR] Error handling request
pulp-pulp_content-2 | 2024-04-15T21:05:39.593885724Z Traceback (most recent call last):
pulp-pulp_content-2 | 2024-04-15T21:05:39.593921657Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 518, in thread_handler
pulp-pulp_content-2 | 2024-04-15T21:05:39.593936583Z raise exc_info[1]
pulp-pulp_content-2 | 2024-04-15T21:05:39.593950331Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 518, in thread_handler
pulp-pulp_content-2 | 2024-04-15T21:05:39.593964038Z raise exc_info[1]
pulp-pulp_content-2 | 2024-04-15T21:05:39.593977328Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 619, in _match_and_stream
pulp-pulp_content-2 | 2024-04-15T21:05:39.593991155Z await publication.published_artifact.aget(relative_path=index_path)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594004615Z File “/usr/local/lib/python3.9/site-packages/django/db/models/query.py”, line 649, in aget
pulp-pulp_content-2 | 2024-04-15T21:05:39.594018292Z return await sync_to_async(self.get)(*args, **kwargs)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594031662Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 468, in call
pulp-pulp_content-2 | 2024-04-15T21:05:39.594045249Z ret = await asyncio.shield(exec_coro)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594058582Z File “/usr/lib64/python3.9/concurrent/futures/thread.py”, line 58, in run
pulp-pulp_content-2 | 2024-04-15T21:05:39.594072089Z result = self.fn(*self.args, **self.kwargs)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594085385Z File “/usr/local/lib/python3.9/site-packages/asgiref/sync.py”, line 522, in thread_handler
pulp-pulp_content-2 | 2024-04-15T21:05:39.594101865Z return func(*args, **kwargs)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594115178Z File “/usr/local/lib/python3.9/site-packages/django/db/models/query.py”, line 637, in get
pulp-pulp_content-2 | 2024-04-15T21:05:39.594128842Z raise self.model.DoesNotExist(
pulp-pulp_content-2 | 2024-04-15T21:05:39.594142028Z pulpcore.app.models.publication.PublishedArtifact.DoesNotExist: PublishedArtifact matching query does not exist.
pulp-pulp_content-2 | 2024-04-15T21:05:39.594155641Z
pulp-pulp_content-2 | 2024-04-15T21:05:39.594168727Z During handling of the above exception, another exception occurred:
pulp-pulp_content-2 | 2024-04-15T21:05:39.594182000Z
pulp-pulp_content-2 | 2024-04-15T21:05:39.594195240Z Traceback (most recent call last):
pulp-pulp_content-2 | 2024-04-15T21:05:39.594208643Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py”, line 452, in _handle_request
pulp-pulp_content-2 | 2024-04-15T21:05:39.594222474Z resp = await request_handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594235734Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py”, line 543, in _handle
pulp-pulp_content-2 | 2024-04-15T21:05:39.594249417Z resp = await handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594262587Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py”, line 114, in impl
pulp-pulp_content-2 | 2024-04-15T21:05:39.594276327Z return await handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594289513Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py”, line 48, in authenticate
pulp-pulp_content-2 | 2024-04-15T21:05:39.594303214Z return await handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594316446Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/instrumentation.py”, line 230, in middleware
pulp-pulp_content-2 | 2024-04-15T21:05:39.594350215Z resp = await handler(request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594364456Z File “/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py”, line 352, in cached_function
pulp-pulp_content-2 | 2024-04-15T21:05:39.594378317Z response = await self.make_entry(
pulp-pulp_content-2 | 2024-04-15T21:05:39.594391784Z File “/usr/local/lib/python3.9/site-packages/pulpcore/cache/cache.py”, line 395, in make_entry
pulp-pulp_content-2 | 2024-04-15T21:05:39.594405618Z response = await handler(*args, **kwargs)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594418922Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 257, in stream_content
pulp-pulp_content-2 | 2024-04-15T21:05:39.594432605Z return await self._match_and_stream(path, request)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594446108Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 636, in _match_and_stream
pulp-pulp_content-2 | 2024-04-15T21:05:39.594459809Z body=self.render_html(
pulp-pulp_content-2 | 2024-04-15T21:05:39.594473018Z File “/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py”, line 438, in render_html
pulp-pulp_content-2 | 2024-04-15T21:05:39.594486666Z return template.render(
pulp-pulp_content-2 | 2024-04-15T21:05:39.594499875Z File “/usr/local/lib/python3.9/site-packages/jinja2/environment.py”, line 1301, in render
pulp-pulp_content-2 | 2024-04-15T21:05:39.594513643Z self.environment.handle_exception()
pulp-pulp_content-2 | 2024-04-15T21:05:39.594527249Z File “/usr/local/lib/python3.9/site-packages/jinja2/environment.py”, line 936, in handle_exception
pulp-pulp_content-2 | 2024-04-15T21:05:39.594541338Z raise rewrite_traceback_stack(source=source)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594554687Z File “”, line 15, in top-level template code
pulp-pulp_content-2 | 2024-04-15T21:05:39.594568722Z File “/usr/local/lib/python3.9/site-packages/jinja2/filters.py”, line 683, in do_filesizeformat
pulp-pulp_content-2 | 2024-04-15T21:05:39.594582516Z bytes = float(value)
pulp-pulp_content-2 | 2024-04-15T21:05:39.594595729Z ValueError: could not convert string to float: ‘None\n’
pulp-pulp_content-2 | 2024-04-15T21:05:39.594309233Z ::ffff:172.27.0.12 [15/Apr/2024:21:05:39 +0000] “GET /pulp/content/PyPI/ HTTP/1.0” 500 315 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36”
If I change the policy to immediate:
Task fails with the following error:
“error”: {
“traceback”: " File “/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py”, line 70, in _execute_task\n result = func(*args, **kwargs)\n File “/usr/local/lib/python3.9/site-packages/pulp_python/app/tasks/sync.py”, line 61, in sync\n DeclarativeVersion(first_stage, repository, mirror).create()\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py”, line 161, in create\n loop.run_until_complete(pipeline)\n File “/usr/lib64/python3.9/asyncio/base_events.py”, line 647, in run_until_complete\n return future.result()\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py”, line 220, in create_pipeline\n await asyncio.gather(*futures)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py”, line 41, in call\n await self.run()\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 186, in run\n pb.done += task.result() # download_count\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py”, line 241, in _handle_content_unit\n await asyncio.gather(*downloaders_for_content)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/models.py”, line 119, in download\n raise e\n File “/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/models.py”, line 111, in download\n download_result = await downloader.run(extra_data=self.extra_data)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 269, in run\n return await download_wrapper()\n File “/usr/local/lib/python3.9/site-packages/backoff/_async.py”, line 151, in retry\n ret = await target(*args, **kwargs)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 254, in download_wrapper\n return await self._run(extra_data=extra_data)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 290, in _run\n self.raise_for_status(response)\n File “/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py”, line 187, in raise_for_status\n response.raise_for_status()\n File “/usr/local/lib64/python3.9/site-packages/aiohttp/client_reqrep.py”, line 1060, in raise_for_status\n raise ClientResponseError(\n",
“description”: “403, message=‘InternalAVError’, url=URL(‘https://files.pythonhosted.org/packages/21/33/4925decd863ce88ed9190a4bd872b01c146243ee68db08c72923984fe335/torch-2.0.1-cp310-cp310-manylinux2014_aarch64.whl’)”
},
Not sure why we are getting 403 from upstream for some of the packages. I can see that some are being downloaded successfully though.