Hello All
We are running two POC Pulp3 environments.
I’ll have to explain.
The environment:
- Single container ± Two years running CI/CD all fine.
- Minimal-api/content/worker four months. Also fine with an exception.
Both POC instances uses the same Ansible playbook to create
six repositories via CI/CD.
A repo form SLES (smt) is defined as a remote Pulp3 repo.
From that a local upstream Pulp3 repo is built.
This happens daily to collect the daily updates from SMT.
SLES12-SP3/4/5 and SLES15-SP3/4/5
From each SP level a single repository is built. That combines
for each SP’s individual module, e.g. SLES15SP5 have 33 SMT SLES repos,
one large repository.
e.g. 2023Q03-SLES15-SP5-latest & 2023Q03-SLES12-SP5-latest etc.
The copy API used: http://pulp/pulp/api/v3/rpm/copy/
With the option to NOT do dependency_solving.
dependency_solving: false
This API copy is done per upstream relevant repo.
This process is sub 6 seconds per upstream repository copy.
Really very quick and uses no filesystem space.
It is used as a Zypper repository, also very effective.
The problem:
It was noticed some time ago on the more attention getting Pulp mimimal POC that:
This API copy process is all of a sudden taking much longer
on the SLES15-SP5-latest repository.
From the sub 6 second time per task to copy the upstream to 2023Q03-SLES15-SP5-latest.
To +5 minutes per upstream repository. 5*32 minutes.
The CI/CD pipeline is taking ± 5Minutes longer every day as well.
From a 23 minute run per day we are up to 220 minutes a run and increasing daily.
==
So this week I had time to look into this.
The Single container is using Pulp Core pulp/pulp:3.31 and the API copy is fast.
Changing this to Pulp core pulp/pulp:3.32 introduced the slow API copy.
The Pulp minimal image /pulp/pulp-minimal:3.38 is very slow on the 2nd POC.
This is where the CI/CD time to compete was noticed.
Changing to this version today from 3.38:
/pulp/pulp-minimal:3.31 pulp-worker
/pulp/pulp-minimal:3.31 pulp-api
/pulp/pulp-minimal:3.31 pulp-content
The API copy time came down the sub 6 seconds again.
And the repository looks the same.
You can probably guess the question.
What changed in Pulp 3.32 on-wards that causes this huge delay in the API copy?
It could be that I missed an announcement regarding this.
If so, please refer me.
The reason it is asked this way is the the Ansible used in both POCs are the same.
Unchanged for the last three years.
Expected outcome: API copy to behave the same as in Pulp 3.31.
Note:
The other 5 repositories copy times are still short. (OMW)
Thanks so much for the assistance.
Expected outcome:
API copy to behave the same as in Pulp 3.31
Pulpcore version:
Version 3.32 and above
Pulp plugins installed and their versions:
Operating system - distribution and version:
Sles15 SP5
Other relevant data:
Sample task show output.
pulp task show --href "/pulp/api/v3/tasks/018b6b8e-3931-7bd7-92b2-71aa4cfe1e73/"
{
"pulp_href": "/pulp/api/v3/tasks/018b6b8e-3931-7bd7-92b2-71aa4cfe1e73/",
"pulp_created": "2023-10-26T12:36:04.786338+02:00",
"state": "running",
"name": "pulp_rpm.app.tasks.copy.copy_content",
"logging_cid": "7f375560c29544c7b7e58effc6327c83",
"created_by": "/pulp/api/v3/users/1/",
"started_at": "2023-10-26T12:36:04.907917+02:00",
"finished_at": null,
"error": null,
"worker": "/pulp/api/v3/workers/018b6b86-ad80-75c0-9798-4f1091818601/",
"parent_task": null,
"child_tasks": [],
"task_group": null,
"progress_reports": [],
"created_resources": [],
"reserved_resources_record": [
"/pulp/api/v3/repositories/rpm/rpm/6f6e8805-5977-486f-91a9-f10809f69682/",
"shared:/pulp/api/v3/repositories/rpm/rpm/3bef47ce-9b12-4ea5-9abe-5c0b308a1e14/",
"shared:/pulp/api/v3/domains/21aa1298-c509-4bdf-9fcd-fe917fb6b5c9/"
]
}
The API call in the playbook:
---
- name: Add Backend repos to MR repo
uri:
url: "{{ PULP_SVR }}{{ COPY_API }}"
method: POST
body_format: json
validate_certs: no
user: admin
password: "{{ PASSWD }}"
force_basic_auth: yes
body:
config:
- source_repo_version: "{{ item }}"
dest_repo: "{{ NEW_DST }}"
dependency_solving: false
status_code: 202
loop: "{{ SRC_REPO }}"
loop_control:
index_var: a_idx
register: add_result
---
The SRC_REPO:
TASK [Add Backend repos to MR repo] ********************************************
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/3bef47ce-9b12-4ea5-9abe-5c0b308a1e14/versions/7/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/91cb198d-9168-4ccc-b58a-dc27ba2ba5a6/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/e4b7f797-51b6-4c45-9def-5e09afdff186/versions/82/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/0a3a6019-102d-45c4-b69b-30b99040303e/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/74da5357-0654-418d-8010-9ef764a9d307/versions/18/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/d6cd730a-38a1-4a3a-9e25-fea02530d54f/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/af1fc045-69a7-4aae-b900-a1426cabfa72/versions/44/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/c0345375-7a41-47d9-a243-24bfb260b92d/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/d7ccbfd6-7194-4a57-a1b1-e06e65b5ed5a/versions/53/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/2d1bfaf7-5d42-4855-927c-3a058e7b4f8d/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/e84cd7db-a0b3-4726-9e77-a125c98b4d11/versions/23/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/7cf7739e-222c-4951-8c8f-483113eb6204/versions/5/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/f6e02e6e-afae-4a5d-8b9e-3913e9bbeddf/versions/52/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/bfb6d223-3e73-4a3f-9a35-3b10b9f2c7c8/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/b231ee04-7ad9-4f97-8675-b4bb8a87692e/versions/27/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/47c3fac1-9dc3-4d51-8f0d-29f2f9649954/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/b44d614f-b4e2-4fe0-92a3-39de327529c9/versions/8/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/dd27a1e4-b33c-4cd7-b7ef-77d596abc9bc/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/a4882df0-8195-4aeb-a072-0d450c66861e/versions/14/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/b9039f28-cbaf-4fbf-84a1-b72d97b9ffe1/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/8e773e06-9d87-4b64-a9e3-e0c4359fa577/versions/38/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/15aaa5d6-02b3-47a7-908b-7529505cf56b/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/49387c74-94f7-4917-be2e-d32fc14ae23a/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/f7919683-9553-43e4-97fb-038c49438b85/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/5433ddd2-274a-483b-b494-b9d38940a0c4/versions/8/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/394997a2-fcab-423e-a2cb-444f86240346/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/40a8902d-a49c-44b7-a101-f12aab291d3f/versions/17/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/879b1571-314f-4341-a43e-4fb8c0b70f58/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/48fa7bd6-cd1b-4ef9-9ea5-b70f29bcbbc0/versions/1/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/e54c773d-9ba4-46ca-a2dd-d42f24564e79/versions/4/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/fb389646-532b-4cf0-a2d7-9c91df8bf648/versions/0/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/55fc1250-4397-4c8b-aa68-eea7e56b8a31/versions/3/)
ok: [localhost] => (item=/pulp/api/v3/repositories/rpm/rpm/3a5a0cdf-8831-4c28-a850-cd8b91b17155/versions/47/)