More bugs? unable to download certain packages

[bkwfanena]

Problem:
Unable to download packages / NGINX errors.

Expected outcome:
should be able to download packages

Pulpcore version:
versions:

• component: core
  package: pulpcore
  version: 3.22.4
• component: rpm
  package: pulp-rpm
  version: 3.19.4
• component: container
  package: pulp-container
  version: 2.14.5

Pulp plugins installed and their versions:

Operating system - distribution and version:
RHEL-7

** LOGS **
raise Exception(_(“Expected path ‘{}’ is not found”).format(path))

Exception: Expected path ‘/var/lib/pulp/media/artifact/ea/103ff86b8ff2d7de92b1042fba36ddca4c13e3b712e63d1980115bc91dda5b’ is not found

127.0.0.1 [16/May/2023:23:35:56 +0000] “GET /pulp/content/ol-7/x86_64/ol7_latest/20230517/Packages/b/bind-utils-9.11.4-16.P2.el7_8.6.x86_64.rpm HTTP/1.0” 500 225 “https://localhost/pulp/content/ol-7/x86_64/ol7_latest/20230517/Packages/b/” “Wget/1.14 (linux-gnu)”

[2023-05-16 23:35:56 +0000] [12245] [ERROR] Error handling request

Traceback (most recent call last):

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/aiohttp/web_protocol.py”, line 433, in _handle_request

resp = await request_handler(request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/aiohttp/web_app.py”, line 504, in _handle

resp = await handler(request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/aiohttp/web_middlewares.py”, line 117, in impl

return await handler(request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/content/authentication.py”, line 41, in authenticate

return await handler(request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/cache/cache.py”, line 339, in cached_function

response = await self.make_entry(

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/cache/cache.py”, line 378, in make_entry

response = await handler(*args, **kwargs)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/content/handler.py”, line 243, in stream_content

return await self._match_and_stream(path, request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/content/handler.py”, line 573, in _match_and_stream

return await self._serve_content_artifact(ca, headers, request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/content/handler.py”, line 847, in _serve_content_artifact

raise Exception(_(“Expected path ‘{}’ is not found”).format(path))

Exception: Expected path ‘/var/lib/pulp/media/artifact/00/01cdc86487e0b94cbd56449c7d270a5b4710b29071ec80f9f73b47c9f6a42e’ is not found

127.0.0.1 [16/May/2023:23:35:56 +0000] “GET /pulp/content/ol-7/x86_64/ol7_latest/20230517/Packages/b/bind-utils-9.11.4-26.P2.el7.x86_64.rpm HTTP/1.0” 500 225 “https://localhost/pulp/content/ol-7/x86_64/ol7_latest/20230517/Packages/b/” “Wget/1.14 (linux-gnu)”

[2023-05-16 23:35:56 +0000] [12254] [ERROR] Error handling request

Traceback (most recent call last):

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/aiohttp/web_protocol.py”, line 433, in _handle_request

resp = await request_handler(request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/aiohttp/web_app.py”, line 504, in _handle

resp = await handler(request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/aiohttp/web_middlewares.py”, line 117, in impl

return await handler(request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/content/authentication.py”, line 41, in authenticate

return await handler(request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/cache/cache.py”, line 339, in cached_function

response = await self.make_entry(

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/cache/cache.py”, line 378, in make_entry

response = await handler(*args, **kwargs)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/content/handler.py”, line 243, in stream_content

return await self._match_and_stream(path, request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/content/handler.py”, line 573, in _match_and_stream

return await self._serve_content_artifact(ca, headers, request)

File “/usr/local/lib/pulp/lib64/python3.8/site-packages/pulpcore/content/handler.py”, line 847, in _serve_content_artifact

raise Exception(_(“Expected path ‘{}’ is not found”).format(path))

Exception: Expected path ‘/var/lib/pulp/media/artifact/e5/70eb0b7ad51056d77381a99be5576a17fa743fc8d2baf5c5fb8494186c98c1’ is not found

127.0.0.1 [16/May/2023:23:35:56 +0000] “GET /pulp/content/ol-7/x86_64/ol7_latest/20230517/Packages/b/bind-utils-9.11.4-26.P2.el7_9.10.x86_64.rpm HTTP/1.0” 500 225 “https://localhost/pulp/content/ol-7/x86_64/ol7_latest/20230517/Packages/b/” “Wget/1.14 (linux-gnu)”

[2023-05-16 23:35:56 +0000] [12254] [ERROR] Error handling request

Traceback (most recent call last):

The above error happens after a fresh repo is created and synced
commands used to create and sync the repo is as below

pulp rpm remote create --name test_ol_7 --url https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/ --policy immediate --tls-validation false
pulp rpm repository create --name test_ol_7  --remote test_ol_7  --gpgcheck 0 --autopublish --repo-gpgcheck 0
pulp rpm distribution create --name test_ol_7 --base-path test/ol7/live --repository test_ol_7

[ggainey]

Hey there bkwfanena,

First - I don’t see your ‘sync’ call in your example. Would help to post it, so we can be sure we’re looking at the same thing.

Second - I can’t reproduce the problem today. Used your cmds to create/sync from ORA7, and was able to wget the bind-rpm without a problem.

Third, and concerning - the bind-utils-9.11.4-26.P2.el7_9.10.x86_64.rpm RPM downloaded from ORA’s site has a SHA256 of e570eb0b7ad51056d77381a99be5576a17fa743fc8d2baf5c5fb8494186c98c1 . I would expect its Artifact to be found at /var/lib/pulp/media/artifact/e5/70eb0b7ad51056d77381a99be5576a17fa743fc8d2baf5c5fb8494186c98c1 - and in fact, on my system, it is. The SHA1 checksum of the artifact on my system, matches the SHA1 of the RPM downloaded directly from ORA, and matches the expected SHA1 from the primary.xml.gz in the ORA repodata as well.

Right now, your system is a) looking for an artifact whose 256sum doesn’t match that of the content you’re seeking, and b) that Artifact appears to have been removed from your artifact-storage. I don’t know how that happened - but that’s the proximate cause of the error you’re getting, the system data is in an inconsistent state.

You might try a ‘repair’ on that repository - checkout out the doc for that workflow Do let us know what your sync-call looks like as well!

[bkwfanena]

thanks @ggainey ,
looked like the issue was with missing or corrupted files on the filesystem.
I found the doc last night and tried and it seems to have fixed the problem. still not sure how it happened though

name: pulpcore.app.tasks.repository.repair_all_artifacts
parent_task: null
progress_reports:
- code: repair.missing
  done: 2369
  message: Identify missing units
  state: completed
  suffix: null
  total: null
- code: repair.corrupted
  done: 0
  message: Identify corrupted units
  state: completed
  suffix: null
  total: null
- code: repair.repaired
  done: 2369
  message: Repair corrupted units
  state: completed
  suffix: null
  total: null

Please close this issue

[ggainey]

Woohoo! Glad we got you fixed up!

That is…definitely a lot of units repaired. I honestly have no idea what could have left you in that state. We have absolutely had cases where a user accidentally did a rm -rf pulp/artifacts/* when they didn’t mean to, but this feels more random than that.

Anyway - again, glad you’re up and going, you know where to find us igf/when/as you have any more issues.