Pulp repo 404 while downloading few packages

#1

Hi Team ,

I am using pulp operator in openshift and repositories/publications and distributions are created using squeezer . Recently while installing a package I noticed that package while installing is showing as below.

Dependencies resolved.

Package Architecture Version Repository Size

Installing:
zenity x86_64 3.28.1-2.el8 appstream 4.0 M
Installing dependencies:
libnotify x86_64 0.7.7-6.el8 appstream 47 k

Transaction Summary

Install 2 Packages

Total size: 4.0 M
Total download size: 4.0 M
Installed size: 6.3 M
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] libnotify-0.7.7-6.el8.x86_64.rpm: Already downloaded
[MIRROR] zenity-3.28.1-2.el8.x86_64.rpm: Status code: 404 for https://pulpserver/pulp/content/test/rhel8/appstream/Packages/z/zenity-3.28.1-2.el8.x86_64.rpm (IP: 172.28.78.183)
[MIRROR] zenity-3.28.1-2.el8.x86_64.rpm: Status code: 404 for https://pulpserver/pulp/content/test/rhel8/appstream/Packages/z/zenity-3.28.1-2.el8.x86_64.rpm (IP: 172.28.78.183)
[MIRROR] zenity-3.28.1-2.el8.x86_64.rpm: Status code: 404 for https://pulpserver/pulp/content/test/rhel8/appstream/Packages/z/zenity-3.28.1-2.el8.x86_64.rpm (IP: 172.28.78.183)

I could see zenity rpm in GUI , upon clicking it shows 404. Also curl also shows the rpm
[root@yc3293.danskenet.net TEST:~]# curl https://https://pulpserver/pulp/content/test/rhel8/appstream/Packages/z/

Index of /pulp/content/test/rhel8/appstream/Packages/z/

Index of /pulp/content/test/rhel8/appstream/Packages/z/

../
zenity-3.28.1-1.el8.x86_64.rpm                                                                      04-Sep-2024 12:03  4.2 MB
zenity-3.28.1-2.el8.x86_64.rpm                                                                      04-Sep-2024 12:04  4.2 MB
zsh-html-5.5.1-10.el8.noarch.rpm                                                                    04-Sep-2024 12:04  532.3 kB
zsh-html-5.5.1-6.el8_1.2.noarch.rpm                                                                 04-Sep-2024 12:03  532.1 kB
zsh-html-5.5.1-9.el8.noarch.rpm                                                                     04-Sep-2024 12:04  532.2 kB
zstd-1.4.2-2.el8.x86_64.rpm                                                                         04-Sep-2024 12:03  394.0 kB
zstd-1.4.4-1.el8.x86_64.rpm                                                                         04-Sep-2024 12:03  402.8 kB
zziplib-0.13.68-13.el8_10.i686.rpm                                                                  04-Sep-2024 12:05  95.9 kB
zziplib-0.13.68-13.el8_10.x86_64.rpm                                                                04-Sep-2024 12:05  93.6 kB
zziplib-0.13.68-8.el8.i686.rpm                                                                      04-Sep-2024 12:03  95.2 kB
zziplib-0.13.68-8.el8.x86_64.rpm                                                                    04-Sep-2024 12:03  92.9 kB
zziplib-0.13.68-9.el8.i686.rpm                                                                      04-Sep-2024 12:04  95.4 kB
zziplib-0.13.68-9.el8.x86_64.rpm                                                                    04-Sep-2024 12:04  93.1 kB
zziplib-utils-0.13.68-13.el8_10.x86_64.rpm                                                          04-Sep-2024 12:05  49.1 kB
zziplib-utils-0.13.68-8.el8.x86_64.rpm                                                              04-Sep-2024 12:03  48.6 kB
zziplib-utils-0.13.68-9.el8.x86_64.rpm                                                              04-Sep-2024 12:04  48.7 kB

Upon checking I could see there are many packages which shows 404 , means they are not really available .

Could it be that sync/distribution did not go well here ? Any suggestion on how to rectify this ?

FYI : we sync only once in a month from redhat.

#2

These are current versions

“versions”: [
{
“component”: “core”,
“version”: “3.58.0”,
“package”: “pulpcore”,
“module”: “pulpcore.app”,
“domain_compatible”: true
},
{
“component”: “ansible”,
“version”: “0.22.1”,
“package”: “pulp-ansible”,
“module”: “pulp_ansible.app”,
“domain_compatible”: false
},
{
“component”: “container”,
“version”: “2.21.0”,
“package”: “pulp-container”,
“module”: “pulp_container.app”,
“domain_compatible”: false
},
{
“component”: “deb”,
“version”: “3.3.1”,
“package”: “pulp_deb”,
“module”: “pulp_deb.app”,
“domain_compatible”: false
},
{
“component”: “maven”,
“version”: “0.8.1”,
“package”: “pulp-maven”,
“module”: “pulp_maven.app”,
“domain_compatible”: false
},
{
“component”: “ostree”,
“version”: “2.4.3”,
“package”: “pulp-ostree”,
“module”: “pulp_ostree.app”,
“domain_compatible”: true
},
{
“component”: “python”,
“version”: “3.12.3”,
“package”: “pulp-python”,
“module”: “pulp_python.app”,
“domain_compatible”: true
},
{
“component”: “rpm”,
“version”: “3.27.1”,
“package”: “pulp-rpm”,
“module”: “pulp_rpm.app”,
“domain_compatible”: true
},
{
“component”: “certguard”,
“version”: “3.58.0”,
“package”: “pulpcore”,
“module”: “pulp_certguard.app”,
“domain_compatible”: true
},
{
“component”: “file”,
“version”: “3.58.0”,
“package”: “pulpcore”,
“module”: “pulp_file.app”,
“domain_compatible”: true
}
],

#3

Can you show us the details of the Remote used to sync this repository? (Not your auth-info, of course)

This feels like an on-demand sync that worked, but when asking for the actual packages, the URL used at sync-time is failing with a 404. Seeing the data for the Remote that was used will help us figure out what’s going on here.

#4

Log-errors from around the failing GET timestamp would also be useful.

#5

Remote :

[
{
“pulp_href”: “/pulp/api/v3/remotes/rpm/rpm/0191bce6-ba08-7ebd-823e-6a16e6e60bbd/”,
“prn”: “prn:rpm.rpmremote:0191bce6-ba08-7ebd-823e-6a16e6e60bbd”,
“pulp_created”: “2024-09-04T11:58:43.209086Z”,
“pulp_last_updated”: “2024-11-05T23:04:08.394560Z”,
“name”: “remote-appstream-rhel8”,
“url”: “https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/”,
“ca_cert”: “cert_file”,
“client_cert”: “client_cert”,
“tls_validation”: true,
“proxy_url”: “http://proxy:80”,
“pulp_labels”: {},
“download_concurrency”: null,
“max_retries”: null,
“policy”: “on_demand”,
“total_timeout”: null,
“connect_timeout”: null,
“sock_connect_timeout”: null,
“sock_read_timeout”: null,
“headers”: null,
“rate_limit”: null,
“hidden_fields”: [
{
“name”: “client_key”,
“is_set”: true
},
{
“name”: “proxy_username”,
“is_set”: false
},
{
“name”: “proxy_password”,
“is_set”: false
},
{
“name”: “username”,
“is_set”: false
},
{
“name”: “password”,
“is_set”: false
}
],
“sles_auth_token”: null
}
]

#6

Any idea what was changed for the remote yesterday? “pulp_last_updated”: “2024-11-05T23:04:08.394560Z”,

#7

I am using squeezer module to sync all cdn remote every night . However publications will be created only once in a month . While syncing repo , it checks for repo/remote existence and squeezer module updates remote every time while repository remains same.

TASK [2vlx-awx-pulp-repo : Create “remote-appstream-rhel9” repository] *********00:03:59
changed: [localhost]

TASK [2vlx-awx-pulp-repo : Create “remote-appstream-rhel8” repository] *********00:04:05
changed: [localhost]

#8

Also , I am using a month old publiction for non prod repo and 2 month old publication for prod repo. Have not noticed this issue before atleast .

So as per my logic , sync the repo every day and create publication once in a month and distribute it to non prod and move current non prod distribtion to prod .

#9

we have an issue with redhat subscriptions where in current certs are not working. We are woking to get new certs. Could it be the reason for 404 ?

#10

If that’s on-demand content, and your certs don’t work, then that would definitely be the problem. Your client asks for the RPM, the content-app notices the binary isn’t there yet, it opens a connection to cdn.redhat.com to ask for it and is told “403” (iirc), and tells the client “I can’t find that binary” with a 404.

#11

Thanks for the insight . subscription is fixed and its working now . Will change to immediate mode here after .

2 Likes