API call: /pulp/api/v3/rpm/copy/ takes longer every day

Problem:

The API /pulp/api/v3/rpm/copy/ process that took max 15 minutes now runs +1.2 - 4h 25m 25s and increasing every day. Small repo is 1.2 hr and the big one +4 hr.

The suspicion is it started with pulp-minimal:3.68.
But is now running pulp-minimal:3.69.

REPOSITORY                TAG         IMAGE ID
pulp/pulp-minimal         3.69        6468ff4fbc35

Was away the whole of December and installed it last week.(3.69)

No real data are being copied, as you know.

    - name: Add Backend repos to MR repo
      uri:
        url: "{{ PULP_SVR }}/pulp/api/v3/rpm/copy/"
        method: POST
        body_format: json
        validate_certs: no
        user: admin
        password: "{{ PASSWD }}"
        force_basic_auth: yes
        body:
          config:
            - source_repo_version: "{{ item }}"
              dest_repo:  "{{ NEW_DST }}"
          dependency_solving: false 
        status_code: 202
      loop: "{{ SRC_REPO }}"
      loop_control:
        index_var: a_idx
      register: add_result

Expected outcome:

Copy process to complete quickly again.

Thank you for the help.

Pulpcore version:

      "versions": {
        "deb": "3.4.0",
        "rpm": "3.27.2",
        "core": "3.69.0",
        "file": "3.69.0",
        "maven": "0.8.1",
        "ostree": "2.4.4",
        "python": "3.12.5",
        "ansible": "0.23.1",
        "certguard": "3.69.0",
        "container": "2.22.0"
      }

Pulp plugins installed and their versions:

Operating system - distribution and version:

SLES15 SP5

Other relevant data:

Even 15 minutes is probably much too long to say nothing of 4 hours, but could you provide some context as to how much content is being copied around, how many repository versions you have, how frequently this is occurring, etc?

Are you able to do a bit of profiling using the TASK_DIAGNOSTICS setting with pyinstrument, to show where the time is being spent? Diagnostics - Pulp Project

What does

      loop: "{{ SRC_REPO }}"
      loop_control:
        index_var: a_idx
      register: add_result

refer to?

A Repository latest are created form the source repos.
There are thousands of files.

The source is used in the loop.

"retain_repo_versions": null,
 "retain_package_versions": 3,

We have had source repos with larger version than these.

One cannot use squeezer for repo versions.
I can use the API to change (think I will do that)

On which of the three pulp minimal/containers is pyinstrument to be installed.
Do I understand correctly, it will start logging as soon as it is installed?

  - name: "active-P-2024Q04.00-BMW-SLES15-SP6-latest"
    base: "MR/SLES15/SP6/2024Q04.00/active-P-2024Q04.00-BMW-SLES15-SP6-latest"
    source:
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-537d-7b2a-9577-a6c875d4b2da/versions/10/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-58af-71c1-99db-a2003e4f21fd/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-5db2-7234-9f95-83faa2673c21/versions/57/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-632a-7266-bace-63a8e88be6ad/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-685f-73c2-920e-b89ab59a2645/versions/17/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-6d87-75ee-a9ca-eca6556da890/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-72c0-7d33-891b-3bdbc7685c43/versions/25/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-77b7-7ec8-a4f2-c1cf226ac74f/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-7ca6-787a-8d89-834fee5b0012/versions/38/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-81a2-7d6f-821e-aa3f3a0dfe32/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-86a6-7f7a-a955-f9118cd42511/versions/19/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-8bdd-7b59-a894-a60d16564a67/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-90e0-7fcd-b2bf-bdb8085e6cbe/versions/48/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-95d0-7c71-ab5a-8ed18edead2e/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-9b63-7ab6-b1cd-3759cb0e52f0/versions/14/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-a040-789a-86ba-5f591b55ec2a/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-a535-7db3-9bd7-02d9edebe393/versions/20/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-aa5d-70dd-aacc-4de274050133/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-af82-7df1-9552-ab9c9085ebe4/versions/5/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-b4cb-7029-82f2-737c77b4830a/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-b9c1-7c4d-b075-e96d27e81adc/versions/32/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-bec0-771b-aebb-b038bbf59ddd/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15b-c3b4-71d3-9792-11aeb70fa0f5/versions/4/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-a7b3-7f2e-a025-ab2278ae8ec9/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-ace6-7495-b109-1768d08099ed/versions/7/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-b1ff-7b2a-a647-66042a0a18d8/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-b6f5-79ca-a311-23998ebb55ae/versions/18/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-bc0c-7603-97e8-0bf1f6cb49fd/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-c152-7ab3-96e9-508ffb73e056/versions/3/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-ca00-7425-be85-0d458c1c08f2/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-d290-774f-8540-5c03a0a567b7/versions/0/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-dc6f-7734-86c3-210c22c69e39/versions/1/"
      - "/pulp/api/v3/repositories/rpm/rpm/0191e15d-e6bf-7fb3-aef9-aa2bd9e696b7/versions/35/"
      ```

Thank you.