Pulp Python Repository does not sync behind Proxy - again

Problem:
Python Repository, when synced with Python PyPI remote, always stays in version 0, i.e. no packages are added.

Our Pulp node is behind a HTTP proxy.

This issue was reported before:

And resolved as part of “python”: “3.12.1”. But it has come back again.

Expected outcome:
The repository version increases after syncs, and packages get added to the repository.

Pulpcore version:
3.75.0

Pulp plugins installed and their versions:
“versions”: {
“deb”: “3.5.1”,
“npm”: “0.3.2”,
“rpm”: “3.29.0”,
“core”: “3.75.0”,
“file”: “3.75.0”,
“maven”: “0.10.0”,
“ostree”: “2.4.6”,
"python": “3.13.2”,
“ansible”: “0.24.3”,
“certguard”: “3.75.0”,
“container”: “2.24.1”
}

Operating system - distribution and version:
NAME=“Alma Linux”
VERSION=“8.9”

Other relevant data:
$ pulp python remote list
[
{
“pulp_href”: “/pulp/api/v3/remotes/python/python/019601ed-f52b-7d93-a34e-d3d7ae7e2b79/”,
“prn”: “prn:python.pythonremote:019601ed-f52b-7d93-a34e-d3d7ae7e2b79”,
“pulp_created”: “2025-04-04T17:51:34.188686Z”,
“pulp_last_updated”: “2025-04-04T17:51:34.188704Z”,
“name”: “TEST-PyPI”,
“url”: “https://pypi-dot-org/”,
“ca_cert”: null,
“client_cert”: null,
"tls_validation": false,
"proxy_url": “http://X.X.X.X:8080/”,
“pulp_labels”: {
“OS”: “PyPI”
},
“download_concurrency”: 10,
“max_retries”: null,
"policy": “immediate”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: false
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“includes”: [
“absl-py==2.1.0”,
“addict==2.4.0”,
“aiofiles==22.1.0”,
“aiohttp-retry==2.8.3”,
“aiohttp==3.9.5”,
“aiosignal==1.3.1”
],
“excludes”: [],
“prereleases”: true,
“package_types”: [
“sdist”,
“bdist_wheel”
],
“keep_latest_packages”: 0,
“exclude_platforms”: [
“freebsd”,
“macos”
]
}
]

Sync task shows no artifact has been downloaded, but no errors either:
{
“pulp_href”: “/pulp/api/v3/tasks/019601ee-0a4e-7990-b421-8c106a439593/”,
“prn”: “prn:core.task:019601ee-0a4e-7990-b421-8c106a439593”,
“pulp_created”: “2025-04-04T17:51:39.599607Z”,
“pulp_last_updated”: “2025-04-04T17:51:39.599635Z”,
"state": “completed”,
“name”: “pulp_python.app.tasks.sync.sync”,
“logging_cid”: “9460bf807d594bc28b609f3185f3509d”,
“created_by”: “/pulp/api/v3/users/1/”,
“unblocked_at”: “2025-04-04T17:51:39.621305Z”,
“started_at”: “2025-04-04T17:51:39.712351Z”,
“finished_at”: “2025-04-04T17:57:55.062394Z”,
“error”: null,
“worker”: “/pulp/api/v3/workers/019601ec-751c-75b0-b70e-45d2e0d65a9c/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [
{
“message”: “Fetching Project Metadata”,
“code”: “sync.fetching.project”,
“state”: “completed”,
“total”: null,
“done”: 0,
“suffix”: null
},
{
“message”: “Downloading Artifacts”,
“code”: “sync.downloading.artifacts”,
“state”: “completed”,
"total": null,
** “done”: 0,**
“suffix”: null
},
{
“message”: “Associating Content”,
“code”: “associating.content”,
“state”: “completed”,
"total": null,
** “done”: 0,**
“suffix”: null
}
],
“created_resources”: [],
“reserved_resources_record”: [
“prn:python.pythonrepository:019601ed-e754-7fb3-bf0e-e78edfc76583”,
“shared:prn:python.pythonremote:019601ed-f52b-7d93-a34e-d3d7ae7e2b79”,
“shared:prn:core.domain:62823342-753d-4170-b485-8259710b2f00”
]
}

This issue was reported before:

And resolved as part of “python”: “3.12.1”. But it has come back again.

Just to clarify, for you did the fix in 3.12.1 correct the proxy sync issue and now on upgrade you are seeing a regression? Can you do a pip list inside the container and post the results?

Correct, pulp python 3.12.1 resolved the issue that was originally raised by us regarding this.

Our latest update 3.13.2 has the problem again, no python packages are synced in.

pip list inside the container:

bash-5.1$ pip list
Package Version


aiodns 3.2.0
aiofiles 24.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.11.16
aiohttp_socks 0.10.1
aiohttp-xmlrpc 1.5.0
aiosignal 1.3.2
annotated-types 0.7.0
ansible-builder 3.1.0
ansible-compat 24.10.0
ansible-core 2.14.18
ansible-lint 6.22.2
asgiref 3.8.1
async-lru 2.0.5
async-timeout 4.0.3
asyncio-throttle 1.0.2
attrs 22.2.0
azure-core 1.33.0
azure-storage-blob 12.25.1
backoff 2.2.1
bandersnatch 6.3.0
beautifulsoup4 4.13.3
bindep 2.13.0
black 25.1.0
boto3 1.37.27
botocore 1.37.27
bracex 2.5.post1
cachetools 5.5.2
certifi 2025.1.31
cffi 1.14.5
chardet 5.2.0
charset-normalizer 3.4.1
click 8.1.8
createrepo_c 1.2.1
cryptography 44.0.2
dbus-python 1.2.18
defusedxml 0.7.1
Deprecated 1.2.18
diff-match-patch 20241021
distro 1.9.0
Django 4.2.20
django-auth-ldap 5.1.0
django-filter 25.1
django-guid 3.5.1
django-import-export 3.3.9
django-lifecycle 1.2.4
django-readonly-field 1.1.2
django-storages 1.14.5
djangorestframework 3.15.2
djangorestframework-queryfields 1.1.0
drf-access-policy 1.5.0
drf-nested-routers 0.94.1
drf-spectacular 0.27.2
dynaconf 3.2.10
et_xmlfile 2.0.0
filelock 3.18.0
flake8 6.1.0
frozenlist 1.5.0
galaxy_importer 0.4.29
gitdb 4.0.12
GitPython 3.1.44
google-api-core 2.24.2
google-auth 2.38.0
google-cloud-core 2.4.3
google-cloud-storage 3.1.0
google-crc32c 1.7.1
google-resumable-media 2.7.2
googleapis-common-protos 1.69.2
gpg 1.15.1
gunicorn 23.0.0
humanfriendly 10.0
idna 3.10
importlib-metadata 6.0.1
importlib_resources 6.4.5
inflection 0.5.1
iniparse 0.5
isodate 0.7.2
Jinja2 3.1.6
jmespath 1.0.1
jq 1.8.0
json-stream 2.3.3
json-stream-rs-tokenizer 0.4.29
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
libcomps 0.1.21.post1
lxml 5.3.1
mailbits 0.2.2
Mako 1.1.4.dev0
Markdown 3.3.4
markdown-it-py 3.0.0
MarkupPy 1.18
MarkupSafe 3.0.2
mccabe 0.7.0
mdurl 0.1.2
multidict 6.2.0
mypy-extensions 1.0.0
nh3 0.2.21
odfpy 1.4.1
openpyxl 3.1.5
opentelemetry-api 1.31.1
opentelemetry-exporter-otlp-proto-common 1.31.1
opentelemetry-exporter-otlp-proto-http 1.31.1
opentelemetry-proto 1.31.1
opentelemetry-sdk 1.31.1
opentelemetry-semantic-conventions 0.52b1
packaging 24.2
Parsley 1.3
pathspec 0.12.1
pbr 6.1.1
pillow 11.1.0
pip 21.3.1
pkginfo 1.12.1.2
platformdirs 4.3.7
ply 3.11
productmd 1.33
propcache 0.3.1
proto-plus 1.26.1
protobuf 5.29.4
psycopg 3.2.6
psycopg-binary 3.2.6
psycopg2 2.8.6
pulp-ansible 0.24.3
pulp-certguard 1.8.0
pulp-container 2.24.1
pulp-deb 3.5.1
pulp-file 1.16.0
pulp-glue 0.32.0
pulp-maven 0.10.0
pulp-npm 0.3.2
pulp-ostree 2.4.6
pulp-python 3.13.2
pulp-rpm 3.29.0
pulpcore 3.75.0
pyasn1 0.6.1
pyasn1_modules 0.4.2
pycairo 1.27.0
pycares 4.5.0
pycodestyle 2.11.1
pycparser 2.20
pydantic 2.11.2
pydantic_core 2.33.1
pyflakes 3.1.0
Pygments 2.19.1
PyGObject 3.40.1
pygtrie 2.5.0
PyJWT 2.10.1
pyOpenSSL 25.0.0
pyparsing 3.2.3
pypi-simple 1.6.1
python-dateutil 2.8.1
python-debian 0.1.49
python-gnupg 0.5.4
python-ldap 3.4.4
python-nginx 1.5.7
python-socks 2.7.1
PyYAML 6.0.2
redis 5.2.1
referencing 0.36.2
requests 2.32.3
resolvelib 0.8.1
rhsm 1.19.2
rich 14.0.0
rpds-py 0.24.0
rpm 4.16.1.3
rsa 4.9
ruamel.yaml 0.18.10
ruamel.yaml.clib 0.2.12
s3transfer 0.11.4
semantic-version 2.10.0
setuptools 53.0.0
six 1.15.0
smmap 5.0.2
solv 0.7.29
soupsieve 2.6
sqlparse 0.5.3
subprocess-tee 0.4.2
systemd-python 234
tablib 3.5.0
tomli 2.2.1
typing_extensions 4.13.1
typing-inspection 0.4.0
uritemplate 4.1.1
url-normalize 2.2.0
urllib3 1.26.20
uuid6 2024.7.10
wcmatch 10.0
whitenoise 6.9.0
wrapt 1.17.2
xlrd 2.0.1
xlwt 1.3.0
yamllint 1.37.0
yarl 1.18.3
zipp 3.21.0

Please note that similar to last time, the task does not fail, but 0 packages are actually downloaded, as shown in the task logs.

@Pulp-User-333 I think I found the regression, if you have the ability to do so I would appreciate testing out the fix: Fix proxy-sync regression by gerrod3 · Pull Request #824 · pulp/pulp_python · GitHub

3 Likes

Sure. I can test it once the new images become available on DockerHub, do you have a timeframe for this? And thanks.

I think gerrod is asking if you have the opportunity to apply that patch to your system and check that it actually fixes your issue, before we merge and build containers with it :slight_smile: Not everyone can do that, of course - but it gives us a lot more confidence that the PR is fixing the right thing, when we don’t have a local reproducer.

Noted. I just injected the updated sync.py into the Pulp worker container, and the packages are now being synced successfully. So the change has solved the issue. Thanks!

Do you know the timeframe when we can have the new updated Pulp images with this fix included?

1 Like

If I could ask a favor of you - if you could put the above, into a comment on Gerrod’s PR, that would get things moving. Once a PR is approved and merged, releases can happen really quickly!

1 Like

Done.

3 Likes

pulp_python 3.13.3 is out now, it should be included in the images later tonight.

3 Likes

Perfect, thanks.

1 Like