Pulp Python Repository does not sync behind Proxy

Problem:

Python Repository, when synced with Python PyPI remote, always stays in version 0, i.e. no packages are added.

Our Pulp node is behind a HTTP proxy.

Expected outcome:

The repository version increases after syncs, and packages get added to the repository.

Pulpcore version:
{
“component”: “core”,
“version”: “3.43.1”,
“package”: “pulpcore”,
“domain_compatible”: true
},

Pulp plugins installed and their versions:
“versions”: [

{
  "component": "ansible",
  "version": "0.21.1",
  "package": "pulp-ansible",
  "domain_compatible": false
},
{
  "component": "certguard",
  "version": "1.7.1",
  "package": "pulp-certguard",
  "domain_compatible": true
},
{
  "component": "container",
  "version": "2.17.0",
  "package": "pulp-container",
  "domain_compatible": false
},
{
  "component": "deb",
  "version": "3.1.1",
  "package": "pulp_deb",
  "domain_compatible": false
},
{
  "component": "maven",
  "version": "0.8.0",
  "package": "pulp-maven",
  "domain_compatible": false
},
{
  "component": "ostree",
  "version": "2.2.1",
  "package": "pulp-ostree",
  "domain_compatible": false
},
{
  "component": "python",
  "version": "3.11.0",
  "package": "pulp-python",
  "domain_compatible": false
},
{
  "component": "rpm",
  "version": "3.24.0",
  "package": "pulp-rpm",
  "domain_compatible": true
},
{
  "component": "file",
  "version": "3.43.1",
  "package": "pulp-file",
  "domain_compatible": true
}

]

Operating system - distribution and version:
NAME=“Alma Linux”
VERSION=“8.9”

Other relevant data:

$ pulp python remote list
[
{
“pulp_href”: “/pulp/api/v3/remotes/python/python/018e3b8e-4db9-7318-9c09-0786781fa9ee/”,
“pulp_created”: “2024-03-14T06:02:45.563065Z”,
“name”: “Test-PyPI”,
“url”: “https://pypi.org/”,
“ca_cert”: null,
“client_cert”: null,
“tls_validation”: false,
“proxy_url”: “http://10.1.1.9:8080”,
“pulp_labels”: {},
“pulp_last_updated”: “2024-03-14T06:02:45.563083Z”,
“download_concurrency”: null,
“max_retries”: null,
“policy”: “immediate”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: false
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“includes”: [
“numpy”,
“shelf-reader”
],
“excludes”: [],
“prereleases”: false,
“package_types”: [],
“keep_latest_packages”: 0,
“exclude_platforms”: []
}
]

1 Like

I don’t have a full answer, but I can provide some additional context: I recently tested exactly this workflow (syncing some python content using an HTTP proxy), and it work for me. I was using the following versions:

python3.11-pulp-python-3.10.0-3.el8.x86_64.rpm
python3.11-pulpcore-3.39.11-1.el8.x86_64.rpm
python3.11-aiohttp-3.9.2-1.el8.x86_64.rpm

This is on a Katello 4.11.1 installation.

I wonder if the Python 3.11 and aiohttp version might be relevant here, since I know there were some changes regarding HTTP proxies.

1 Like

Does your proxy allow you to create tls connections even though the connection to the proxy might not be encrypted? I think it should be safe to keep tls_validation on.

It is just a HTTP proxy, not HTTPs.

aiohttp-3.9.1 is also installed for our Pulp node.

pulp-python == 3.11.0
pulpcore == 3.43.1

Yes, but proxies usually still allow creating ssl/tls connections to far away servers.
Actually there may be more types of http proxies out there than I can think of.

Can specify this with more details?

Oh, I see what you mean. Yes, I can make HTTPS requests to remote hosts via the proxy. However, the HTTP proxy kind of messes with the cert chain, and makes it invalid. Hence, I disable the TLS validation. I think this is equivalent of “pip install –trusted-host server-name package”.

1 Like

Can you paste the task output for the sync?

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

Looking at the Fetching Project Metadata report we see it processed 0 packages despite the remote having two listed in the includes. This suggests it couldn’t reach PyPI. Can you paste your logs here for this sync? There should be messages detailing what HTTP error message it ran into.

We have updated remote.includes to have only one package listed:

“includes”: [
“shelf-reader”
]

pulp python repository sync --name Test-PyPI --remote Test-PyPI

Logs:
pulp-pulp_web-1 | 2024-03-16T03:07:52.364468307Z 192.168.64.1 - admin [16/Mar/2024:03:07:52 +0000] “GET /pulp/api/v3/repositories/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.1” 200 564 “-” "Pulp-CLI/0.24.0"pulp-pulp_api-2 | 2024-03-16T03:07:52.365079753Z (‘pulp [b46b2674aaec47c4a1c02fd71890c6aa]: ::ffff:192.168.64.12 - admin [16/Mar/2024:03:07:52 +0000] “GET /pulp/api/v3/repositories/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.0” 200 564 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-16T03:07:52.907162151Z 192.168.64.1 - admin [16/Mar/2024:03:07:52 +0000] “GET /pulp/api/v3/remotes/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.1” 200 921 “-” “Pulp-CLI/0.24.0”
pulp-pulp_api-1 | 2024-03-16T03:07:52.907699904Z (‘pulp [b46b2674aaec47c4a1c02fd71890c6aa]: ::ffff:192.168.64.12 - admin [16/Mar/2024:03:07:52 +0000] “GET /pulp/api/v3/remotes/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.0” 200 921 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_api-2 | 2024-03-16T03:07:53.467830868Z (‘pulp [b46b2674aaec47c4a1c02fd71890c6aa]: ::ffff:192.168.64.12 - admin [16/Mar/2024:03:07:53 +0000] “POST /pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/sync/ HTTP/1.0” 202 67 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-16T03:07:53.467953830Z 192.168.64.1 - admin [16/Mar/2024:03:07:53 +0000] “POST /pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/sync/ HTTP/1.1” 202 67 “-” “Pulp-CLI/0.24.0”
pulp-pulp_worker-4 | 2024-03-16T03:07:53.547767927Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: pulpcore.tasking.tasks:INFO: Starting task 018e453a-eceb-7f99-89fd-a3ca6789e1d8
pulp-pulp_worker-4 | 2024-03-16T03:07:53.630862591Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized project plugin allowlist_project, filtering [‘shelf-reader’]
pulp-pulp_worker-4 | 2024-03-16T03:07:53.645181546Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized release plugin allowlist_release, filtering [<Requirement(‘shelf-reader’)>]
pulp-pulp_worker-4 | 2024-03-16T03:07:53.646854518Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized release plugin blocklist_release, filtering []
pulp-pulp_worker-4 | 2024-03-16T03:07:53.652540995Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized prerelease plugin with [re.compile(’.+rc\d+$’), re.compile(’.+a(lpha)?\d+$’), re.compile(’.+b(eta)?\d+$’), re.compile(’.+dev\d+$’)]
pulp-pulp_worker-4 | 2024-03-16T03:07:53.652572418Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch:INFO: Initialized prerelease plugin prerelease_release, filtering all packages
pulp-pulp_worker-4 | 2024-03-16T03:07:53.661182674Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch.mirror:INFO: Syncing with <>
pulp-pulp_worker-4 | 2024-03-16T03:07:53.661248245Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch.mirror:INFO: No metadata filters are enabled. Skipping metadata filtering
pulp-pulp_worker-4 | 2024-03-16T03:07:53.661403643Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch.mirror:INFO: No release file filters are enabled. Skipping release file filtering
pulp-pulp_worker-4 | 2024-03-16T03:07:53.661863283Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: bandersnatch.package:INFO: Fetching metadata for package: shelf-reader (serial 0)
pulp-pulp_worker-4 | 2024-03-16T03:07:53.664973114Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: aiohttp.client:WARNING: Could not read .netrc file: [Errno 2] No such file or directory: ‘.fake-netrc’
pulp-pulp_worker-4 | 2024-03-16T03:07:53.665557572Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: aiohttp.client:WARNING: Could not read .netrc file: [Errno 2] No such file or directory: ‘.fake-netrc’
pulp-pulp_worker-4 | 2024-03-16T03:07:53.860220482Z pulp [b46b2674aaec47c4a1c02fd71890c6aa]: pulpcore.tasking.tasks:INFO: Task completed 018e453a-eceb-7f99-89fd-a3ca6789e1d8
pulp-pulp_web-1 | 2024-03-16T03:07:54.045535571Z 192.168.64.1 - admin [16/Mar/2024:03:07:54 +0000] “GET /pulp/api/v3/tasks/018e453a-eceb-7f99-89fd-a3ca6789e1d8/ HTTP/1.1” 200 1166 “-” “Pulp-CLI/0.24.0”
pulp-pulp_api-1 | 2024-03-16T03:07:54.046062910Z (‘pulp [b46b2674aaec47c4a1c02fd71890c6aa]: ::ffff:192.168.64.12 - admin [16/Mar/2024:03:07:54 +0000] “GET /pulp/api/v3/tasks/018e453a-eceb-7f99-89fd-a3ca6789e1d8/ HTTP/1.0” 200 1166 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-16T03:07:54.065133558Z 2024/03/16 03:07:54 [info] 29#0: *20633 client 192.168.64.1 closed keepalive connection

=============================================

The error: [Errno 2] No such file or directory: ‘.fake-netrc’ above is, I believe, intentional implementation in Pulp:
"
# Prevent bandersnatch from reading actual .netrc file, set to nonexistent file
environ[“NETRC”] = f"{path.curdir}/.fake-netrc"
"

Also, as pasted in the previous reply, the task does not report any error either:

pulp task show --uuid 018e453a-eceb-7f99-89fd-a3ca6789e1d8
{
“pulp_href”: “/pulp/api/v3/tasks/018e453a-eceb-7f99-89fd-a3ca6789e1d8/”,
“pulp_created”: “2024-03-16T03:07:53.452019Z”,
“state”: “completed”,
“name”: “pulp_python.app.tasks.sync.sync”,
“logging_cid”: “b46b2674aaec47c4a1c02fd71890c6aa”,
“created_by”: “/pulp/api/v3/users/1/”,
“started_at”: “2024-03-16T03:07:53.542987Z”,
“finished_at”: “2024-03-16T03:07:53.858502Z”,
"error": null,
“worker”: “/pulp/api/v3/workers/018e3f22-d719-77d7-a538-c53dd68c0cd8/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [
{
“message”: “Fetching Project Metadata”,
“code”: “sync.fetching.project”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
},
{
“message”: “Downloading Artifacts”,
“code”: “sync.downloading.artifacts”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
},
{
“message”: “Associating Content”,
“code”: “associating.content”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
}
],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/”,
“shared:/pulp/api/v3/remotes/python/python/018e3f85-b903-783a-a05a-f86fdabce839/”,
“shared:/pulp/api/v3/domains/018e3f21-814e-7d37-ad03-074f05743a5f/”
]
}

Thank you

Sadly those log messages were not that helpful, probably because we currently consume the error and do nothing with it: https://github.com/pulp/pulp_python/blob/main/pulp_python/app/tasks/sync.py#L255-L260. If you can edit these lines and add a logging statement here, we will get a better understanding of what the error is.

Another thing after doing some testing myself, the proxy-url format is specific. aiohttp expects it to be in the format scheme://host:port, e.g. http://127.0.0.1:8899. In your proxy’s logs you should see one request to PyPI per package listed in includes.

I made sure the remote proxy url includes the scheme (http).

pulp python remote list

[
{
“pulp_href”: “/pulp/api/v3/remotes/python/python/018e3f85-b903-783a-a05a-f86fdabce839/”,
“pulp_created”: “2024-03-15T00:31:52.068466Z”,
“name”: “Test-PyPI”,
"url": “https://pypi.org/”,
“ca_cert”: null,
“client_cert”: null,
"tls_validation": false,
"proxy_url": “http://10.1.1.9:8080”,
“pulp_labels”: {},
“pulp_last_updated”: “2024-03-18T15:58:31.957989Z”,
“download_concurrency”: null,
“max_retries”: null,
“policy”: “on_demand”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: false
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“includes”: [
“shelf-reader”
],
“excludes”: [],
“prereleases”: true,
“package_types”: [],
“keep_latest_packages”: 0,
“exclude_platforms”: []
}
]

Here are the sync logs when I added the logging of the exception:

pulp python repository sync --name Test-PyPI --remote Test-PyPI

pulp-pulp_web-1 | 2024-03-18T21:47:10.652786064Z 192.168.176.1 - admin [18/Mar/2024:21:47:10 +0000] “GET /pulp/api/v3/repositories/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.1” 200 564 “-” “Pulp-CLI/0.24.0”
pulp-pulp_api-1 | 2024-03-18T21:47:10.653551158Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:10 +0000] “GET /pulp/api/v3/repositories/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.0” 200 564 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:47:11.746604115Z 192.168.176.1 - admin [18/Mar/2024:21:47:11 +0000] “GET /pulp/api/v3/remotes/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.1” 200 927 “-” “Pulp-CLI/0.24.0”
pulp-pulp_api-1 | 2024-03-18T21:47:11.747284862Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:11 +0000] “GET /pulp/api/v3/remotes/python/python/?name=Test-PyPI&offset=0&limit=1 HTTP/1.0” 200 927 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_api-1 | 2024-03-18T21:47:12.324803857Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:12 +0000] “POST /pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/sync/ HTTP/1.0” 202 67 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:47:12.324410264Z 192.168.176.1 - admin [18/Mar/2024:21:47:12 +0000] “POST /pulp/api/v3/repositories/python/python/018e3f85-bfde-7228-b620-68f8f49897aa/sync/ HTTP/1.1” 202 67 “-” “Pulp-CLI/0.24.0”
pulp-pulp_worker-1 | 2024-03-18T21:47:12.418490029Z pulp [3603bba994724b058c5d25aa67d47796]: pulpcore.tasking.tasks:INFO: Starting task 018e5388-682c-7f27-9c78-4732adb78743
pulp-pulp_worker-1 | 2024-03-18T21:47:12.498437946Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch:INFO: Initialized project plugin allowlist_project, filtering [‘shelf-reader’]
pulp-pulp_worker-1 | 2024-03-18T21:47:12.514885930Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch:INFO: Initialized release plugin allowlist_release, filtering [<Requirement(‘shelf-reader’)>]
pulp-pulp_worker-1 | 2024-03-18T21:47:12.516635901Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch:INFO: Initialized release plugin blocklist_release, filtering []
pulp-pulp_worker-1 | 2024-03-18T21:47:12.529957467Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch.mirror:INFO: Syncing with https:[slash][slash]pypi[dot]org.
pulp-pulp_worker-1 | 2024-03-18T21:47:12.529986932Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch.mirror:INFO: No metadata filters are enabled. Skipping metadata filtering
pulp-pulp_worker-1 | 2024-03-18T21:47:12.530143699Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch.mirror:INFO: No release file filters are enabled. Skipping release file filtering
pulp-pulp_worker-1 | 2024-03-18T21:47:12.530519436Z pulp [3603bba994724b058c5d25aa67d47796]: bandersnatch.package:INFO: Fetching metadata for package: shelf-reader (serial 0)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.533371593Z pulp [3603bba994724b058c5d25aa67d47796]: aiohttp.client:WARNING: Could not read .netrc file: [Errno 2] No such file or directory: ‘.fake-netrc’
pulp-pulp_worker-1 | 2024-03-18T21:47:12.533914395Z pulp [3603bba994724b058c5d25aa67d47796]: aiohttp.client:WARNING: Could not read .netrc file: [Errno 2] No such file or directory: ‘.fake-netrc’
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763058486Z pulp [3603bba994724b058c5d25aa67d47796]: pulp_python.app.tasks.sync:ERROR: Cannot connect to host pypi[dot]org[colon]443 ssl:True [SSLCertVerificationError: (1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)’)]
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763096450Z Traceback (most recent call last):
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763111835Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 1104, in _start_tls_connection
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763126207Z tls_transport = await self._loop.start_tls(
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763139747Z File “/usr/lib64/python3.9/asyncio/base_events.py”, line 1240, in start_tls
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763153561Z await waiter
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763167128Z File “/usr/lib64/python3.9/asyncio/sslproto.py”, line 534, in data_received
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763180892Z ssldata, appdata = self._sslpipe.feed_ssldata(data)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763221799Z File “/usr/lib64/python3.9/asyncio/sslproto.py”, line 188, in feed_ssldata
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763236276Z self._sslobj.do_handshake()
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763249648Z File “/usr/lib64/python3.9/ssl.py”, line 945, in do_handshake
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763263279Z self._sslobj.do_handshake()
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763276656Z ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763290399Z
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763303602Z The above exception was the direct cause of the following exception:
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763317029Z
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763330008Z Traceback (most recent call last):
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763343360Z File “/usr/local/lib/python3.9/site-packages/bandersnatch/mirror.py”, line 129, in package_syncer
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763357181Z await package.update_metadata(self.master, attempts=3)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763370651Z File “/usr/local/lib/python3.9/site-packages/bandersnatch/package.py”, line 61, in update_metadata
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763384288Z self._metadata = await master.get_package_metadata(
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763397608Z File “/usr/local/lib/python3.9/site-packages/bandersnatch/master.py”, line 220, in get_package_metadata
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763411422Z metadata_response = await metadata_generator.asend(None)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763424789Z File “/usr/local/lib/python3.9/site-packages/bandersnatch/master.py”, line 132, in get
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763438743Z async with self.session.get(path, **kw) as r:
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763452140Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/client.py”, line 1194, in aenter
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763465773Z self._resp = await self._coro
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763478966Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/client.py”, line 578, in _request
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763492563Z conn = await self._connector.connect(
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763507306Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 544, in connect
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763521066Z proto = await self._create_connection(req, traces, timeout)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763534306Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 909, in _create_connection
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763547943Z _, proto = await self._create_proxy_connection(req, traces, timeout)
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763561256Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 1357, in _create_proxy_connection
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763574933Z return await self._start_tls_connection(
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763588173Z File “/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py”, line 1118, in _start_tls_connection
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763601896Z raise ClientConnectorCertificateError(req.connection_key, exc) from exc
pulp-pulp_worker-1 | 2024-03-18T21:47:12.763642774Z aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host pypi[dot]org[colon]443 ssl:True [SSLCertVerificationError: (1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)’)]
pulp-pulp_api-1 | 2024-03-18T21:47:12.887955498Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:12 +0000] “GET /pulp/api/v3/tasks/018e5388-682c-7f27-9c78-4732adb78743/ HTTP/1.0” 200 1139 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:47:12.888969825Z 192.168.176.1 - admin [18/Mar/2024:21:47:12 +0000] “GET /pulp/api/v3/tasks/018e5388-682c-7f27-9c78-4732adb78743/ HTTP/1.1” 200 1139 “-” “Pulp-CLI/0.24.0”
pulp-pulp_worker-1 | 2024-03-18T21:47:12.964190192Z pulp [3603bba994724b058c5d25aa67d47796]: pulpcore.tasking.tasks:INFO: Task completed 018e5388-682c-7f27-9c78-4732adb78743
pulp-pulp_api-1 | 2024-03-18T21:47:14.461346967Z (‘pulp [3603bba994724b058c5d25aa67d47796]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:47:14 +0000] “GET /pulp/api/v3/tasks/018e5388-682c-7f27-9c78-4732adb78743/ HTTP/1.0” 200 1166 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:47:14.462688653Z 192.168.176.1 - admin [18/Mar/2024:21:47:14 +0000] “GET /pulp/api/v3/tasks/018e5388-682c-7f27-9c78-4732adb78743/ HTTP/1.1” 200 1166 “-” “Pulp-CLI/0.24.0”
pulp-pulp_web-1 | 2024-03-18T21:47:14.478233165Z 2024/03/18 21:47:14 [info] 29#0: *1 client 192.168.176.1 closed keepalive connection
^R
remitepulp-pulp_api-1 | 2024-03-18T21:49:30.642575292Z (‘pulp [2a6037aedb5f4a6fa186ad1b50bff8be]: ::ffff:192.168.176.7 - admin [18/Mar/2024:21:49:30 +0000] “GET /pulp/api/v3/remotes/python/python/?offset=0&limit=25 HTTP/1.0” 200 1913 “-” “Pulp-CLI/0.24.0”’,)
pulp-pulp_web-1 | 2024-03-18T21:49:30.642580167Z 192.168.176.1 - admin [18/Mar/2024:21:49:30 +0000] “GET /pulp/api/v3/remotes/python/python/?offset=0&limit=25 HTTP/1.1” 200 1913 “-” “Pulp-CLI/0.24.0”
pulp-pulp_web-1 | 2024-03-18T21:49:30.665551733Z 2024/03/18 21:49:30 [info] 29#0: *8 client 192.168.176.1 closed keepalive connection

From the logs above, it seems like it is still trying to perform TLS verification, despite having it set to False on the remote.

Yes I think you are correct, this is probably a bug in pulp_python. We are not telling bandersnatch (the library we use to get the PyPI metadata) to not perform TLS when it has been disabled on the remote. Can you file an issue for this? https://github.com/pulp/pulp_python/issues/new/choose

3 Likes

Done. Thanks.

2 Likes

Pleas let us know if you need assistance with this bug resolution.

If you want to fix the issue, you would do it in this method here[0].

The instructions for setting up a development environment are here[1].

[0] https://github.com/pulp/pulp_python/blob/main/pulp_python/app/tasks/sync.py#L62
[1] Contributing with code - Pulp Project