Changing "--retain-package versions" parameter is not reflected in my repo

[orelops]

Problem: I want to change the value of this “–retain-package-versions” parameter and set it to 1 to keep only 1 version of my package.

1 - pulp rpm repository update --retain-package-versions 1 --name=foreman-3.2
2 - pulp rpm repository update --name “foreman-3.2” --remote “foreman-3.2”
3 - pulp rpm publication create --repository “foreman-3.2”
4 - pulp rpm distribution update --name “foreman-3.2” --publication “/pulp/api/v3/publications/rpm/rpm/d5a55d4a-740a-47b2-a601-2a4a82c24dde/”

Pulpcore version:
{
“component”: “core”,
“version”: “3.20.0”,
“package”: “pulpcore”
},

Pulp plugins installed and their versions:
{
“component”: “rpm”,
“version”: “3.17.7”,
“package”: “pulp-rpm”
},
{
“component”: “file”,
“version”: “1.10.3”,
“package”: “pulp-file”
},
{
“component”: “container”,
“version”: “2.13.0”,
“package”: “pulp-container”
},
{
“component”: “ansible”,
“version”: “0.14.0”,
“package”: “pulp-ansible”
}

Operating system - distribution and version: Alma 8.6

Other relevant data:

he keeps 2 versions
I forgot something ?

[ipanova]

You need to sync the repo, package retention is being applied on the repository version based on its contents the publication is created.

[orelops]

Hi @ipanova

my steps:

(venv-pulp3) [admin@pulp pulp3]$ pulp rpm repository update --retain-package-versions 1 --name=foreman-3.2
Started background task /pulp/api/v3/tasks/843c2971-e5b8-4006-ba90-bff8d3951500/
Done.
(venv-pulp3) [admin@pulp pulp3]$ pulp rpm repository show --name foreman-3.2
{
“pulp_href”: “/pulp/api/v3/repositories/rpm/rpm/ba57c21f-8bd4-4cd8-90a8-cef951225630/”,
“pulp_created”: “2022-08-08T16:09:09.776712Z”,
“versions_href”: “/pulp/api/v3/repositories/rpm/rpm/ba57c21f-8bd4-4cd8-90a8-cef951225630/versions/”,
“pulp_labels”: {},
“latest_version_href”: “/pulp/api/v3/repositories/rpm/rpm/ba57c21f-8bd4-4cd8-90a8-cef951225630/versions/2/”,
“name”: “foreman-3.2”,
“description”: null,
“retain_repo_versions”: 1,
“remote”: “/pulp/api/v3/remotes/rpm/rpm/73e9cb76-b8ee-42de-90bb-bd107ff53060/”,
“autopublish”: false,
“metadata_signing_service”: null,
“retain_package_versions”: 1,
“metadata_checksum_type”: null,
“package_checksum_type”: null,
“gpgcheck”: 0,
“repo_gpgcheck”: 0,
“sqlite_metadata”: false
}

pulp rpm repository sync --name foreman-3.2
Started background task /pulp/api/v3/tasks/e7c546e4-16b7-4dc9-a634-9dca7b528901/
.Done.
(venv-pulp3) [admin@pulp pulp3]$ pulp rpm publication create --repository foreman-3.2
Started background task /pulp/api/v3/tasks/651b81a7-b5ba-4bbe-b631-a635289ec838/
…Done.
{
“pulp_href”: “/pulp/api/v3/publications/rpm/rpm/279e7a50-1c80-4841-9f96-899099de74b5/”,
“pulp_created”: “2022-08-09T09:56:21.803321Z”,
“repository_version”: “/pulp/api/v3/repositories/rpm/rpm/ba57c21f-8bd4-4cd8-90a8-cef951225630/versions/2/”,
“repository”: “/pulp/api/v3/repositories/rpm/rpm/ba57c21f-8bd4-4cd8-90a8-cef951225630/”,
“metadata_checksum_type”: “sha256”,
“package_checksum_type”: “sha256”,
“gpgcheck”: 0,
“repo_gpgcheck”: 0,
“sqlite_metadata”: false
}

(venv-pulp3) [admin@pulp pulp3]$ pulp rpm distribution update --name foreman-3.2 --publication /pulp/api/v3/publications/rpm/rpm/279e7a50-1c80-4841-9f96-899099de74b5/
Started background task /pulp/api/v3/tasks/fa87d68c-d6ba-43d1-88d3-dc93ef318846/
Done.

[ipanova]

@orelops

latest_version_href

@dralley what to do in this case? mirror mode is incompatible with package retention. New repo version was not created( nothing changed on remote sourse) so the package retention policy has not been applied. Does he need to manually remove the version?

[orelops]

@orelops

pulp rpm distribution update --name foreman-3.2 --publication

in fact, I just tested, at creation if I set the “–retain-package-versions” to 1, it does the job for me but if the repo is already created, it doesn’t work.

[orelops]

as well as if I want to modify the url of my remote and the parameter “–retain-package-versions” is set to 5 for example, at the time of the sync I am blocked with error:

(venv-pulp3) [admin@pulp pulp3]$ pulp rpm repository sync --mirror --name foreman-3.2
Error: [“Cannot use ‘retain_package_versions’ in combination with a ‘mirror_complete’ or ‘mirror_content_only’ sync policy.”]

is there a possible workaround ?

[gerrod]

@orelops Looking through the code retain-package-versions is applied when a new repository version is created, so just updating the field on the repository is not enough. I think you can start a modify task on the repository to apply the new policy. e.g

# Get the latest version
LATEST_VERSION_NUM=pulp rpm repository version show --repository foreman-3.2 | jq -r .number
# Trigger modify task using latest version to create new version w/ new retain policy
pulp rpm repository content modify --repository foreman-3.2 --base-version $LATEST_VERSION_NUM

[dralley]

@orelops If you want to use --retain-package-versions you need to use the additive sync policy. That sync should work.

However we may need to add a condition to the sync task to force the sync to go through if --retain-package-versions has been tweaked https://github.com/pulp/pulp_rpm/blob/main/pulp_rpm/app/tasks/synchronizing.py#L324-L362

@ipanova Do you agree?

[ipanova]

I agree, we need to tweak optimization logic to make this work. Additive sync policy won’t be useful if the repo was already synced without package retention and there were no changes on the remote source either.

[orelops]

ok this way it works :

# Updating repository

export REPO_HREF=$(pulp rpm repository update --retain-package-versions ${RETAIN_PACKAGE} --name=${REPONAME} | jq -r ‘.pulp_href’)

Inspecting Repository

pulp rpm repository show --name “${REPONAME}”

update content of repository

LATEST_VERSION_NUM=$(pulp rpm repository version show --repository ${REPONAME} | jq -r .number)
pulp rpm repository content modify --repository ${REPONAME} --base-version ${LATEST_VERSION_NUM}

Sync repository

echo “Create a task to sync the repository using the remote.”
pulp rpm repository update --name “${REPONAME}” --remote “${REPONAME}”
pulp rpm repository sync --name “${REPONAME}” --remote “${REPONAME}”

Create RPM publication

echo “Create a task to create a publication.”
TASK_HREF=$(pulp rpm publication create --repository “${REPONAME}” 2>&1 >/dev/null | awk ‘{print $4}’)

Updating RPM distribution for publication

echo “Set PUBLICATION_HREF from finished task.”
PUBLICATION_HREF=$(pulp show --href “${TASK_HREF}” | jq -r ‘.created_resources | first’)
export PUBLICATION_HREF

echo “updating the rpm distribtion for publication”
pulp rpm distribution update --name “${REPONAME}” --publication “${PUBLICATION_HREF}”

packets are removed.

Now I did the reverse test by wanting to increase the --retain-package-versions parameter to 5, I run the same script but I notice that the task returns me “Skipping Sync (no change from previous sync)”

(venv-pulp3) [admin@pulp pulp3]$ pulp show --href /pulp/api/v3/tasks/1a6844ad-32d1-4f76-bc06-9a9c33699c09/

{
“pulp_href”: “/pulp/api/v3/tasks/1a6844ad-32d1-4f76-bc06-9a9c33699c09/”,
“pulp_created”: “2022-08-09T16:01:02.156484Z”,
“state”: “completed”,
“name”: “pulp_rpm.app.tasks.synchronizing.synchronize”,
“logging_cid”: “1df70162cb494eaf87544546521c5e19”,
“started_at”: “2022-08-09T16:01:02.199749Z”,
“finished_at”: “2022-08-09T16:01:02.728585Z”,
“error”: null,
“worker”: “/pulp/api/v3/workers/bcccb229-28e8-4f21-898f-e72e3aadddbd/”,
“parent_task”: null,
“child_tasks”: [],
“task_group”: null,
“progress_reports”: [
{
“message”: “Skipping Sync (no change from previous sync)”,
“code”: “sync.was_skipped”,
“state”: “completed”,
“total”: 1,
“done”: 1,
“suffix”: null
}
],
“created_resources”: [],
“reserved_resources_record”: [
“/pulp/api/v3/repositories/rpm/rpm/8eb364d7-9441-4877-b792-6f11eafb3c4d/”,
“shared:/pulp/api/v3/remotes/rpm/rpm/eed02370-3829-4c75-93b6-3781bf50bc39/”
]
}

[gerrod]

@orelops

update content of repository

LATEST_VERSION_NUM=$(pulp rpm repository version show --repository ${REPONAME} | jq -r .number)
pulp rpm repository content modify --repository ${REPONAME} --base-version ${LATEST_VERSION_NUM}

To clarify a bit further: retain-package-versions changes the behavior of the repository when new repository versions are created. The modify & sync tasks both generate a new repository version.

@orelops

Sync repository

echo “Create a task to sync the repository using the remote.”
pulp rpm repository update --name “${REPONAME}” --remote “${REPONAME}”
pulp rpm repository sync --name “${REPONAME}” --remote “${REPONAME}”

A heads up, each repository can have a default remote. When you call sync if you don’t specify a remote it will try to use the default remote, else it will error if another remote isn’t specified on the sync call.

@orelops

Now I did the reverse test by wanting to increase the --retain-package-versions parameter to 5, I run the same script but I notice that the task returns me “Skipping Sync (no change from previous sync)”

This is because RPM syncs are optimized to not do any work if no updates have occurred on the upstream repository. Sadly, the CLI is missing the --optimize field for the sync call, but setting that to False will force the sync to go through, e.g:
http POST "${PULP_API}""${REPO_HREF}"sync/ optimize=False

[orelops]

@gerrod

This is because RPM syncs are optimized to not do any work if no updates have occurred on the upstream repository. Sadly, the CLI is missing the --optimize field for the sync call, but setting that to False will force the sync to go through, e.g:
http POST "${PULP_API}""${REPO_HREF}"sync/ optimize=False

ok thank it works with http

[ipanova]

I filed this RPM issue as a follow up of this thread https://github.com/pulp/pulp_rpm/issues/2704