Pulp Python remote syncs all version of package inclusions despite having keep_latest_packages set to 1

Problem:
Pulp Python remote syncs/downloads all versions of package inclusions despite having keep_latest_packages set to 1

Expected outcome:
Only the latest version of a package is downloaded on sync.
“keep_latest_packages”: 1

Pulpcore version:
{
“component”: “core”,
“version”: “3.54.0”,
“package”: “pulpcore”,
“module”: “pulpcore.app”,
“domain_compatible”: true
}

Pulp plugins installed and their versions:
{
“component”: “ansible”,
“version”: “0.21.6”,
“package”: “pulp-ansible”,
“module”: “pulp_ansible.app”,
“domain_compatible”: false
},
{
“component”: “container”,
“version”: “2.20.0”,
“package”: “pulp-container”,
“module”: “pulp_container.app”,
“domain_compatible”: false
},
{
“component”: “deb”,
“version”: “3.2.0”,
“package”: “pulp_deb”,
“module”: “pulp_deb.app”,
“domain_compatible”: false
},
{
“component”: “maven”,
“version”: “0.8.0”,
“package”: “pulp-maven”,
“module”: “pulp_maven.app”,
“domain_compatible”: false
},
{
“component”: “ostree”,
“version”: “2.3.0”,
“package”: “pulp-ostree”,
“module”: “pulp_ostree.app”,
“domain_compatible”: true
},
{
“component”: “python”,
"version": “3.11.1”,
** “package”: “pulp-python”,**
“module”: “pulp_python.app”,
“domain_compatible”: false
},
{
“component”: “rpm”,
“version”: “3.26.0”,
“package”: “pulp-rpm”,
“module”: “pulp_rpm.app”,
“domain_compatible”: true
},
{
“component”: “certguard”,
“version”: “3.54.0”,
“package”: “pulpcore”,
“module”: “pulp_certguard.app”,
“domain_compatible”: true
},
{
“component”: “file”,
“version”: “3.54.0”,
“package”: “pulpcore”,
“module”: “pulp_file.app”,
“domain_compatible”: true
}

Operating system - distribution and version:
NAME=“Alma Linux”
VERSION=“8.9”
Pulp with docker compose is used for deployment.

Other relevant data:
pulp python remote list:
[
{
“pulp_href”: “/pulp/api/v3/remotes/python/python/019050d3-01c5-7264-b128-2317b27dea57/”,
“pulp_created”: “2024-06-25T19:15:24.230307Z”,
“pulp_last_updated”: “2024-06-25T19:15:24.230332Z”,
“name”: “TEST-PyPI”,
“url”: “https://pypi.org/”,
“ca_cert”: null,
“client_cert”: null,
“tls_validation”: false,
“proxy_url”: “”,
“pulp_labels”: {
“OS”: “PyPI”
},
“download_concurrency”: null,
“max_retries”: null,
“policy”: “immediate”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: false
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“includes”: [
“cython”
],
“excludes”: [],
“prereleases”: false,
“package_types”: [
“bdist_wheel”
],
"keep_latest_packages": 1,
“exclude_platforms”: []
}
]

Here are the downloaded wheels, which are many versions, instead of just the latest version:
Cython-0.17-cp26-cp26mu-manylinux1_x86_64.whl
Cython-3.0.10-cp39-cp39-win32.whl
Cython-3.0.10-cp39-cp39-win_amd64.whl
Cython-3.0.10-py2.py3-none-any.whl

Cython-0.29.9-cp37-cp37m-win32.whl

Cython-3.0.5-cp311-cp311-win32.whl

Cython-3.0.9-cp312-cp312-win_amd64.whl
Cython-3.0.9-cp39-cp39-win_amd64.whl

I reproduced this using your exact remote. Doing some experimentation I actually found that this occurs when using the package_types filter with the keep_latest_packages. If you only use the keep_latest_packages by itself then the filter works properly, only syncing files for the latest release. I need to do some more testing to see if there are more filter mismatches or if this is the sole case.

3 Likes

Thank you for the investigation. Is there any plan to resolve the issue regarding the conflicting filters?

  • package_types
  • keep_latest_packages

Yes, once I’ve done a full investigation for all conflicting filters, I’ll file an issue for it and try to fix it.

3 Likes

Perfect, thanks.

1 Like

Ok, I found the error. Using the package_types filter specifically broke the keep_latest_packages and exclude_platforms filters. Here is the fix: Fix package_types filter breaking others by gerrod3 · Pull Request #692 · pulp/pulp_python · GitHub. Once it is merged I’ll backport and release for pulp_python 3.11 & 3.12.

3 Likes

pulp_python 3.11.2 & 3.12.1 have been released with the fixes. They should show up in our images tomorrow.

3 Likes

Thanks! I will fetch the latest images tomorrow, and test the fix accordingly. Will let you know about the result.

1 Like

Resolution tested successfully! Thanks again.

3 Likes