Sync and Publish Issues with BaseOS Repositories (Rocky/AlmaLinux)

Problem:
I’m experiencing issues with synchronizing and publishing BaseOS repositories for Rocky Linux and AlmaLinux via Pulp. When I attempt to sync the repository with autopublish enabled, it fails with an error related to generating metadata. If I disable autopublish, the sync completes, but a manual publication attempt afterward produces a similar error.

Expected outcome:

  1. Sync with --autopublish:
autopublish
    File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/tasks.py", line 76, in _execute_task
        result = func(*args, **kwargs)
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 568, in synchronize
        repo_version = dv.create() or repo.latest_version()
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
        loop.run_until_complete(pipeline)
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/app/models/repository.py", line 1233, in __exit__
        repository.on_new_version(self)
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/models/repository.py", line 260, in on_new_version
        tasks.publish(
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 400, in publish
        generate_repo_metadata(
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 574, in generate_repo_metadata
        pkg.time_file = repo_pkg_times[package.pk]
  1. Manual publication after sync with --no-autopublish
no-autopublish
File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/tasks.py", line 76, in _execute_task
    result = func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 400, in publish
    generate_repo_metadata(
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 574, in generate_repo_metadata
    pkg.time_file = repo_pkg_times[package.pk]
  1. Cleanup
    Even after performing an pulp orphan cleanup by deleting all content and artifacts from the repository, the error reoccurs on the next sync and publish attempt.

Before that all repositories were synchronized fine, it’s hard for me to determine when everything broke, but most likely after pulpcore update from 3.22 to 3.60, but not a sure thing.

Pulpcore version:

    {
      "component": "core",
      "version": "3.60.0",
      "package": "pulpcore",
      "module": "pulpcore.app",
      "domain_compatible": true
    }

Pulp plugins installed and their versions:

    {
      "component": "python",
      "version": "3.12.3",
      "package": "pulp-python",
      "module": "pulp_python.app",
      "domain_compatible": true
    },
    {
      "component": "rpm",
      "version": "3.27.1",
      "package": "pulp-rpm",
      "module": "pulp_rpm.app",
      "domain_compatible": true
    },
    {
      "component": "certguard",
      "version": "3.60.0",
      "package": "pulpcore",
      "module": "pulp_certguard.app",
      "domain_compatible": true
    },
    {
      "component": "file",
      "version": "3.60.0",
      "package": "pulpcore",
      "module": "pulp_file.app",
      "domain_compatible": true
    },
    {
      "component": "deb",
      "version": "3.4.0",
      "package": "pulp_deb",
      "module": "pulp_deb.app",
      "domain_compatible": false
    },
    {
      "component": "maven",
      "version": "0.8.1",
      "package": "pulp-maven",
      "module": "pulp_maven.app",
      "domain_compatible": false
    }

Operating system - distribution and version:
alma9

Can you give us the error-msgs from the log, and URLs used by the remotes? It’s really hard to debug without data :slight_smile:

sure, sry

Remote rocky, for example:

{
  "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/019324bf-3bac-7f60-a508-8d15787721c2/",
  "pulp_created": "2024-11-13T08:58:52.717370Z",
  "pulp_last_updated": "2024-11-13T08:58:52.717385Z",
  "name": "rpm-remote-rockylinux-9.4-BaseOS-x86_64-os-immed",
  "url": "https://download.rockylinux.org/pub/rocky/9.4/BaseOS/x86_64/os",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": null,
  "pulp_labels": {},
  "download_concurrency": 20,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
  "hidden_fields": [
    {
      "name": "client_key",
      "is_set": false
    },
    {
      "name": "proxy_username",
      "is_set": false
    },
    {
      "name": "proxy_password",
      "is_set": false
    },
    {
      "name": "username",
      "is_set": false
    },
    {
      "name": "password",
      "is_set": false
    }
  ],
  "sles_auth_token": null
}

about the error log, I didn’t get anything in the log except the traceback I provided above.
and the description message of the task itself only includes the id
{ “description”: “UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')”

uuid following

pulp=# select * from core_contentartifact where content_id = '3cdf2d11-6ba3-4e4c-9824-d7683284699f';
-[ RECORD 1 ]-----+-------------------------------------
pulp_id           | 932d601e-69b0-4bd1-9f4d-f7af7f2b1a82
pulp_created      | 2024-08-13 06:06:23.155856+03
pulp_last_updated | 2024-08-13 06:06:23.155862+03
relative_path     | 389-ds-base-2.4.5-9.el9_4.x86_64.rpm
artifact_id       | 6bd5de99-8935-4a81-94e6-4308e6142407
content_id        | 3cdf2d11-6ba3-4e4c-9824-d7683284699f

but this is one example, there are more than one of these packages.

After spending two minutes, I found out that there are only two in total. :slight_smile:

pulp task list --state failed | jq '.. | .error?.description? // empty'                           
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('3cdf2d11-6ba3-4e4c-9824-d7683284699f')"
"UUID('19b9075b-1664-4020-b2ad-0d11428c23c6')"
"UUID('19b9075b-1664-4020-b2ad-0d11428c23c6')"
-[ RECORD 1 ]-----+----------------------------------------------------------------
pulp_id           | 984bb0eb-65bb-4839-9a05-06a025f50235
pulp_created      | 2024-06-01 05:03:54.669465+03
pulp_last_updated | 2024-06-01 05:03:54.66947+03
relative_path     | 389-ds-base-1.4.3.39-3.module+el8.10.0+1754+15117a6d.x86_64.rpm
artifact_id       | d40dc33b-b76d-47b6-bb4f-332fd29e9166
content_id        | 19b9075b-1664-4020-b2ad-0d11428c23c6

heres the second from r8

Um…am I missing something? I don’t see a traceback anywhere - just steps and version-output

in the steps are hidden

  1. Sync with --autopublish:
    File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/tasks.py", line 76, in _execute_task
        result = func(*args, **kwargs)
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 568, in synchronize
        repo_version = dv.create() or repo.latest_version()
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
        loop.run_until_complete(pipeline)
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/app/models/repository.py", line 1233, in __exit__
        repository.on_new_version(self)
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/models/repository.py", line 260, in on_new_version
        tasks.publish(
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 400, in publish
        generate_repo_metadata(
      File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 574, in generate_repo_metadata
        pkg.time_file = repo_pkg_times[package.pk]
  1. Manual publication after sync with --no-autopublish
File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/tasks.py", line 76, in _execute_task
    result = func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 400, in publish
    generate_repo_metadata(
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 574, in generate_repo_metadata
    pkg.time_file = repo_pkg_times[package.pk]

Well, I do feel silly :slight_smile: Is there an error-msg at the head of that traceback, with the actual error msg?

In other news - I sync’d/autopublished Rocky without a problem, using core/ and rpm/latest. Will reset to latest-released and see how that goes.

on the host itself is really just this…

pulp [efa9f173bea141afb1bc5fa3222f4ff3]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=rpm-repo-rocky
pulp [efa9f173bea141afb1bc5fa3222f4ff3]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=rpm-repo-rockylinux
pulp [efa9f173bea141afb1bc5fa3222f4ff3]: pulpcore.tasking.tasks:INFO: Task 019326df-ddbe-76aa-8b1d-d1cd2dcd5e97 failed
pulp [efa9f173bea141afb1bc5fa3222f4ff3]: pulpcore.tasking.tasks:INFO:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 568, in synchronize
    result = func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 400, in publish
    generate_repo_metadata(
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 574, in generate_repo_
    pkg.time_file = repo_pkg_times[package.pk]

debug mode doesn’t give us much more than that.