Squeezer0.1.0: A header you provided implies functionality that is not implemented: Transfer-Encoding

Janr · February 10, 2025, 10:43am

Problem:

The problem is this step is failing 100% of the time without we making changes.
Except to upgrade from pulp_squeezer from 0.0.7 to 0.1.0.
This solved other problems but created this new one.

This is a loop through repositories

Short

msg: 'Task /pulp/api/v3/tasks/0194eec9-277d-7b58-9622-b5d2e6508c6a/ failed: ''An error occurred (NotImplemented) when calling the PutObject operation: A header you provided implies functionality that is not implemented: Transfer-Encoding'''
  p_idx: 0

Long

TASK [Create rpm Puplication of Repositories] **********************************
failed: [localhost] (item={'name': 'active-P-2024Q04.00-ABC-SLES12-SP4-latest', 'base': 'MR/SLES12/SP4/2024Q04.00/active-P-2024Q04.00-ABC-SLES12-SP4-latest', 'source': ['/pulp/api/v3/repositories/rpm/rpm/0191e4b9-a4cd-73aa-bc46-c14fe61b91ae/versions/1/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-aa52-7434-8ac6-bd4a6ddbb3ae/versions/3/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-afd0-7f69-be3f-92fb01fc4870/versions/2/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-b59c-7dbe-a12e-0c177982dc7c/versions/1/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-bc33-7163-bd5c-509341c4ed92/versions/2/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-c21c-7243-bea0-3cfe2ecbbd6c/versions/1/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-c857-7969-84b5-3d309535cf3a/versions/3/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-
...
shortened.


 changed=false 
  ansible_index_var: p_idx
  ansible_loop_var: item
  item:
    base: MR/SLES12/SP4/2024Q04.00/active-P-2024Q04.00-ABC-SLES12-SP4-latest
    name: active-P-2024Q04.00-ABC-SLES12-SP4-latest
    source:
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-a4cd-73aa-bc46-c14fe61b91ae/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-aa52-7434-8ac6-bd4a6ddbb3ae/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-afd0-7f69-be3f-92fb01fc4870/versions/2/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-b59c-7dbe-a12e-0c177982dc7c/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-bc33-7163-bd5c-509341c4ed92/versions/2/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-c21c-7243-bea0-3cfe2ecbbd6c/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-c857-7969-84b5-3d309535cf3a/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-ce3c-7d25-af4f-6dcd0b5e9bf7/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-d416-752b-afb2-66fae42afba4/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-d9d5-7eea-b339-41785ab8aa8e/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-dfaf-7389-b6b7-ba874e380212/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-e58a-7aa8-a64b-d3ec816b7582/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-eb44-7b3e-95f6-c2134877cd5c/versions/2/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-f0de-79fb-9d0b-7605530e933e/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-f66c-7ca7-b988-8da841657ec4/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-fc12-72e6-b1f9-13e5b0ac6344/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4ba-01d2-78f7-b455-52c2234b7da5/versions/5/
  msg: 'Task /pulp/api/v3/tasks/0194eec9-277d-7b58-9622-b5d2e6508c6a/ failed: ''An error occurred (NotImplemented) when calling the PutObject operation: A header you provided implies functionality that is not implemented: Transfer-Encoding'''
  p_idx: 0

PLAY RECAP *********************************************************************
localhost                  : ok=15   changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Expected outcome:

To create publication.

Pulpcore version:

In the CI/CD github pipeline.

# /home/pulpadm/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
pulp.squeezer                            0.1.0


      "versions": {
        "deb": "3.5.0",
        "rpm": "3.27.2",
        "core": "3.69.2",
        "file": "3.69.2",
        "maven": "0.8.2",
        "ostree": "2.4.5",
        "python": "3.13.0",
        "ansible": "0.23.1",
        "certguard": "3.69.2",
        "container": "2.22.1"
      }

Pulp plugins installed and their versions:

# /home/pulpadm/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
pulp.squeezer                            0.1.0

Operating system - distribution and version:

SPES15SP5
Linux 53596c4070c3 5.14.21-150500.55.68-default

Other relevant data:
Problem:

The problem is this step is failing 100% of the time without we making changes.
Except to upgrade from pulp_squeezer from 0.0.7 to 0.1.0.
This solved other problems but created this new one.

This is a loop through repositories

Short

msg: 'Task /pulp/api/v3/tasks/0194eec9-277d-7b58-9622-b5d2e6508c6a/ failed: ''An error occurred (NotImplemented) when calling the PutObject operation: A header you provided implies functionality that is not implemented: Transfer-Encoding'''
  p_idx: 0

Long

TASK [Create rpm Puplication of Repositories] **********************************
failed: [localhost] (item={'name': 'active-P-2024Q04.00-ABC-SLES12-SP4-latest', 'base': 'MR/SLES12/SP4/2024Q04.00/active-P-2024Q04.00-ABC-SLES12-SP4-latest', 'source': ['/pulp/api/v3/repositories/rpm/rpm/0191e4b9-a4cd-73aa-bc46-c14fe61b91ae/versions/1/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-aa52-7434-8ac6-bd4a6ddbb3ae/versions/3/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-afd0-7f69-be3f-92fb01fc4870/versions/2/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-b59c-7dbe-a12e-0c177982dc7c/versions/1/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-bc33-7163-bd5c-509341c4ed92/versions/2/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-c21c-7243-bea0-3cfe2ecbbd6c/versions/1/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-c857-7969-84b5-3d309535cf3a/versions/3/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-ce3c-7d25-af4f-6dcd0b5e9bf7/versions/1/', '/pulp/api/v3/repositories/rpm/rpm/0191e4b9-d416-752b-
changed=false 
  ansible_index_var: p_idx
  ansible_loop_var: item
  item:
    base: MR/SLES12/SP4/2024Q04.00/active-P-2024Q04.00-ABC-SLES12-SP4-latest
    name: active-P-2024Q04.00-ABC-SLES12-SP4-latest
    source:
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-a4cd-73aa-bc46-c14fe61b91ae/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-aa52-7434-8ac6-bd4a6ddbb3ae/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-afd0-7f69-be3f-92fb01fc4870/versions/2/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-b59c-7dbe-a12e-0c177982dc7c/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-bc33-7163-bd5c-509341c4ed92/versions/2/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-c21c-7243-bea0-3cfe2ecbbd6c/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-c857-7969-84b5-3d309535cf3a/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-ce3c-7d25-af4f-6dcd0b5e9bf7/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-d416-752b-afb2-66fae42afba4/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-d9d5-7eea-b339-41785ab8aa8e/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-dfaf-7389-b6b7-ba874e380212/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-e58a-7aa8-a64b-d3ec816b7582/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-eb44-7b3e-95f6-c2134877cd5c/versions/2/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-f0de-79fb-9d0b-7605530e933e/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-f66c-7ca7-b988-8da841657ec4/versions/3/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4b9-fc12-72e6-b1f9-13e5b0ac6344/versions/1/
    - /pulp/api/v3/repositories/rpm/rpm/0191e4ba-01d2-78f7-b455-52c2234b7da5/versions/5/
  msg: 'Task /pulp/api/v3/tasks/0194eec9-277d-7b58-9622-b5d2e6508c6a/ failed: ''An error occurred (NotImplemented) when calling the PutObject operation: A header you provided implies functionality that is not implemented: Transfer-Encoding'''
  p_idx: 0

PLAY RECAP *********************************************************************
localhost                  : ok=15   changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Expected outcome:

To create publication.

Pulpcore version:

In the CI/CD github pipeline.

# /home/pulpadm/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
pulp.squeezer                            0.1.0


      "versions": {
        "deb": "3.5.0",
        "rpm": "3.27.2",
        "core": "3.69.2",
        "file": "3.69.2",
        "maven": "0.8.2",
        "ostree": "2.4.5",
        "python": "3.13.0",
        "ansible": "0.23.1",
        "certguard": "3.69.2",
        "container": "2.22.1"
      }

Pulp plugins installed and their versions:

# /home/pulpadm/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
pulp.squeezer                            0.1.0

Operating system - distribution and version:

SPES15SP5
Linux 53596c4070c3 5.14.21-150500.55.68-default

Other relevant data:

6ec2c4388c77:/alloc/logs/worker_app.stdout.0

2025-02-10T07:42:55.220704006+01:00 stdout F Waiting on postgresql to start...
2025-02-10T07:42:55.220704006+01:00 stdout F Postgres started.
2025-02-10T07:42:55.326926282+01:00 stdout F Checking for database migrations
2025-02-10T07:42:59.209484669+01:00 stdout F Database migrated!
2025-02-10T07:43:01.913312049+01:00 stderr F pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
2025-02-10T07:43:02.267976832+01:00 stderr F pulp [None]: pulpcore.tasking.worker:INFO: New worker '1@6ec2c4388c77' discovered
2025-02-10T07:44:46.013678622+01:00 stderr F pulp [5e01fbfccfe34591962e21adc56beb38]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9a-6a69-7d5b-96b8-25b6bda7f170 in domain: default
2025-02-10T07:44:50.761228256+01:00 stderr F pulp [5e01fbfccfe34591962e21adc56beb38]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9a-6a69-7d5b-96b8-25b6bda7f170 in domain: default
2025-02-10T07:46:56.726666312+01:00 stderr F pulp [ad7a5f583dd24c3dabdcf04fad3a1653]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9c-6a68-7161-b7f4-981205b7fdfc in domain: default
2025-02-10T07:46:56.770391787+01:00 stderr F pulp [ad7a5f583dd24c3dabdcf04fad3a1653]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLES12-SP4-Pool remote=remote_upstream-SLES12-SP4-Pool
2025-02-10T07:46:57.343038589+01:00 stderr F pulp [ad7a5f583dd24c3dabdcf04fad3a1653]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9c-6a68-7161-b7f4-981205b7fdfc in domain: default
2025-02-10T07:46:58.556497977+01:00 stderr F pulp [826f4c65c9b94c8997cb37ea7a53c2d1]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9c-7083-7382-a110-bf3cb9f6e02a in domain: default
2025-02-10T07:46:58.633218876+01:00 stderr F pulp [826f4c65c9b94c8997cb37ea7a53c2d1]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLES12-SP4-Updates remote=remote_upstream-SLES12-SP4-Updates
2025-02-10T07:46:59.270774223+01:00 stderr F pulp [826f4c65c9b94c8997cb37ea7a53c2d1]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9c-7083-7382-a110-bf3cb9f6e02a in domain: default
2025-02-10T07:46:59.803262113+01:00 stderr F pulp [f17fd39615b048f9b7900a2ecf2231b8]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9c-7681-77f0-9341-2f78437e22e1 in domain: default
2025-02-10T07:46:59.850405891+01:00 stderr F pulp [f17fd39615b048f9b7900a2ecf2231b8]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLES12-SP4-LTSS-Updates remote=remote_upstream-SLES12-SP4-LTSS-Updates
2025-02-10T07:47:00.471970118+01:00 stderr F pulp [f17fd39615b048f9b7900a2ecf2231b8]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9c-7681-77f0-9341-2f78437e22e1 in domain: default
2025-02-10T07:47:08.102838886+01:00 stderr F pulp [9515f967c910407295522e1dd7c58a04]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9c-95bf-72d0-9817-cdaeabffbd38 in domain: default
2025-02-10T07:47:08.163939028+01:00 stderr F pulp [9515f967c910407295522e1dd7c58a04]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLE12-SP4-SAP-Pool remote=remote_upstream-SLE12-SP4-SAP-Pool
2025-02-10T07:47:08.678562273+01:00 stderr F pulp [9515f967c910407295522e1dd7c58a04]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9c-95bf-72d0-9817-cdaeabffbd38 in domain: default
2025-02-10T07:47:11.224942464+01:00 stderr F pulp [66bab653613c4ac0bfa812facfaab6a3]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9c-a2cb-7276-86d5-160127ada9f4 in domain: default
2025-02-10T07:47:11.300746180+01:00 stderr F pulp [66bab653613c4ac0bfa812facfaab6a3]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLE-Module-Containers12-Pool remote=remote_upstream-SLE-Module-Containers12-Pool
2025-02-10T07:47:11.858016175+01:00 stderr F pulp [66bab653613c4ac0bfa812facfaab6a3]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9c-a2cb-7276-86d5-160127ada9f4 in domain: default
2025-02-10T07:47:13.127407524+01:00 stderr F pulp [3a73fa9a826d4f5c92bdc3e9eb19ac0c]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9c-a92e-7a7a-9f3a-c4c30f165d2f in domain: default
2025-02-10T07:47:13.173375137+01:00 stderr F pulp [3a73fa9a826d4f5c92bdc3e9eb19ac0c]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLE-Module-Containers12-Updates remote=remote_upstream-SLE-Module-Containers12-Updates
2025-02-10T07:47:13.661567119+01:00 stderr F pulp [3a73fa9a826d4f5c92bdc3e9eb19ac0c]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9c-a92e-7a7a-9f3a-c4c30f165d2f in domain: default
2025-02-10T07:47:14.900141086+01:00 stderr F pulp [2aa278835ee5482da6391605652e9aad]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9c-b022-76d1-b505-00cddefb1245 in domain: default
2025-02-10T07:47:14.962010328+01:00 stderr F pulp [2aa278835ee5482da6391605652e9aad]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLE-Module-Legacy12-Pool remote=remote_upstream-SLE-Module-Legacy12-Pool
2025-02-10T07:47:15.514129138+01:00 stderr F pulp [2aa278835ee5482da6391605652e9aad]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9c-b022-76d1-b505-00cddefb1245 in domain: default
2025-02-10T07:47:17.940310354+01:00 stderr F pulp [867612c3fa0c4b71be84270e929024de]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9c-bcf1-7145-80d6-ac0636394bff in domain: default
2025-02-10T07:47:18.001814972+01:00 stderr F pulp [867612c3fa0c4b71be84270e929024de]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLE-Module-Adv-Systems-Management12-Pool remote=remote_upstream-SLE-Module-Adv-Systems-Management12-Pool
2025-02-10T07:47:18.542301842+01:00 stderr F pulp [867612c3fa0c4b71be84270e929024de]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9c-bcf1-7145-80d6-ac0636394bff in domain: default
2025-02-10T07:48:38.859947397+01:00 stderr F pulp [222fd60cfcf444a79f2075c1d3ec63ab]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9d-f9fd-734f-a2e8-01ae7732cd9b in domain: default
2025-02-10T07:55:35.109643501+01:00 stderr F pulp [222fd60cfcf444a79f2075c1d3ec63ab]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9d-f9fd-734f-a2e8-01ae7732cd9b in domain: default
2025-02-10T07:55:36.055512683+01:00 stderr F pulp [5219f40f2a314554967d51a2e1cbd4b0]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9d-fecc-7c4d-b2d6-f8e5d050c701 in domain: default
2025-02-10T08:00:40.288782959+01:00 stderr F pulp [5219f40f2a314554967d51a2e1cbd4b0]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9d-fecc-7c4d-b2d6-f8e5d050c701 in domain: default
2025-02-10T08:00:41.034961747+01:00 stderr F pulp [492686463b294858b4217a0d766f3067]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-0498-7f4e-9609-3153289cd084 in domain: default
2025-02-10T08:04:03.902969631+01:00 stderr F pulp [492686463b294858b4217a0d766f3067]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-0498-7f4e-9609-3153289cd084 in domain: default
2025-02-10T08:04:04.463438719+01:00 stderr F pulp [7a1babc52824400b953ee524bed206f4]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-0900-75c5-91b8-017051db497d in domain: default
2025-02-10T08:05:31.169834587+01:00 stderr F pulp [7a1babc52824400b953ee524bed206f4]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-0900-75c5-91b8-017051db497d in domain: default
2025-02-10T08:05:31.908924140+01:00 stderr F pulp [3560d4d48fa64481807f142691de0398]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-0ea9-758c-97c7-192bbc648966 in domain: default
2025-02-10T08:09:35.053152991+01:00 stderr F pulp [3560d4d48fa64481807f142691de0398]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-0ea9-758c-97c7-192bbc648966 in domain: default
2025-02-10T08:09:35.891855780+01:00 stderr F pulp [fb00bf59b183426693dd6a9bff70d267]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-1357-7408-ae39-a0e9bcfcf721 in domain: default
2025-02-10T08:11:57.961737976+01:00 stderr F pulp [fb00bf59b183426693dd6a9bff70d267]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-1357-7408-ae39-a0e9bcfcf721 in domain: default
2025-02-10T08:11:58.340214875+01:00 stderr F pulp [7da45cdfdd7549148e8f8f8756515738]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-1775-73e6-a2d7-2383041588c4 in domain: default
2025-02-10T08:15:53.869280819+01:00 stderr F pulp [7da45cdfdd7549148e8f8f8756515738]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-1775-73e6-a2d7-2383041588c4 in domain: default
2025-02-10T08:15:54.451407756+01:00 stderr F pulp [7be4b85d543942f7a5b2af27005a148b]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-1d5f-70b3-bddd-a66559cd41c2 in domain: default
2025-02-10T08:19:29.006456670+01:00 stderr F pulp [7be4b85d543942f7a5b2af27005a148b]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-1d5f-70b3-bddd-a66559cd41c2 in domain: default
2025-02-10T08:19:29.111333429+01:00 stderr F pulp [de9053ca76c943dda8e4dd3c5b97e7e2]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-2307-7994-bffb-a27faa680ada in domain: default
2025-02-10T08:22:08.772352668+01:00 stderr F pulp [de9053ca76c943dda8e4dd3c5b97e7e2]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-2307-7994-bffb-a27faa680ada in domain: default
2025-02-10T08:25:29.728176569+01:00 stderr F pulp [b3f54fd4536e497fa5fadbac5c67f3a6]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-30a9-7e7e-8d8d-8561fb42f3a5 in domain: default
2025-02-10T08:26:31.064212718+01:00 stderr F pulp [b3f54fd4536e497fa5fadbac5c67f3a6]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-30a9-7e7e-8d8d-8561fb42f3a5 in domain: default
2025-02-10T08:26:31.771169882+01:00 stderr F pulp [eb978b438da54e238447a378390974bb]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-3548-7871-885d-f34979711885 in domain: default
2025-02-10T08:31:08.400042424+01:00 stderr F pulp [eb978b438da54e238447a378390974bb]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-3548-7871-885d-f34979711885 in domain: default
2025-02-10T08:31:09.131925939+01:00 stderr F pulp [b96f0f4d5e254e74ad847b502f7ff363]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-39f0-7e59-9ca3-5f25fb280c14 in domain: default
2025-02-10T08:32:41.348702766+01:00 stderr F pulp [b96f0f4d5e254e74ad847b502f7ff363]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-39f0-7e59-9ca3-5f25fb280c14 in domain: default
2025-02-10T08:32:42.026389413+01:00 stderr F pulp [4276d6c9687146b8aa7926251cf888ce]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-3f9e-71ce-8f14-604aef947272 in domain: default
2025-02-10T08:34:21.873923548+01:00 stderr F pulp [4276d6c9687146b8aa7926251cf888ce]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-3f9e-71ce-8f14-604aef947272 in domain: default
2025-02-10T08:34:21.976979995+01:00 stderr F pulp [268da49c28cf49c7b91fdd9bd2785fbc]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-455e-742e-adfa-50cab15a1fc7 in domain: default
2025-02-10T08:35:09.108537988+01:00 stderr F pulp [268da49c28cf49c7b91fdd9bd2785fbc]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-455e-742e-adfa-50cab15a1fc7 in domain: default
2025-02-10T08:35:09.230032735+01:00 stderr F pulp [9cd0a977c96c4e408e7e4b55141d9eef]: pulpcore.tasking.tasks:INFO: Starting task 0194ee9e-4b4a-7e99-95a0-f190033fd8f9 in domain: default
2025-02-10T08:35:45.055106663+01:00 stderr F pulp [9cd0a977c96c4e408e7e4b55141d9eef]: pulpcore.tasking.tasks:INFO: Task completed 0194ee9e-4b4a-7e99-95a0-f190033fd8f9 in domain: default
2025-02-10T08:35:47.983811070+01:00 stderr F pulp [00cdc63c3c7448c0a3b1b8195fc15371]: pulpcore.tasking.tasks:INFO: Starting task 0194eec9-277d-7b58-9622-b5d2e6508c6a in domain: default
2025-02-10T08:35:48.001325010+01:00 stderr F pulp [00cdc63c3c7448c0a3b1b8195fc15371]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=active-P-2024Q04.00-ABC-SLES12-SP4-latest, version=3101
2025-02-10T08:35:48.681693787+01:00 stderr F pulp [None]: botocore.httpchecksum:INFO: Skipping checksum validation. Response did not contain one of the following algorithms: ['crc32', 'sha1', 'sha256'].
2025-02-10T08:35:48.805551169+01:00 stderr F pulp [None]: botocore.httpchecksum:INFO: Skipping checksum validation. Response did not contain one of the following algorithms: ['crc32', 'sha1', 'sha256'].
2025-02-10T08:35:48.881666252+01:00 stderr F pulp [None]: botocore.httpchecksum:INFO: Skipping checksum validation. Response did not contain one of the following algorithms: ['crc32', 'sha1', 'sha256'].
2025-02-10T08:35:48.910536426+01:00 stderr F pulp [None]: botocore.httpchecksum:INFO: Skipping checksum validation. Response did not contain one of the following algorithms: ['crc32', 'sha1', 'sha256'].
2025-02-10T08:35:49.022269523+01:00 stderr F pulp [None]: botocore.httpchecksum:INFO: Skipping checksum validation. Response did not contain one of the following algorithms: ['crc32', 'sha1', 'sha256'].
2025-02-10T08:37:02.296997888+01:00 stderr F pulp [00cdc63c3c7448c0a3b1b8195fc15371]: pulpcore.tasking.tasks:INFO: Task[pulp_rpm.app.tasks.publishing.publish] 0194eec9-277d-7b58-9622-b5d2e6508c6a failed (ClientError: An error occurred (NotImplemented) when calling the PutObject operation: A header you provided implies functionality that is not implemented: Transfer-Encoding) in domain: default
2025-02-10T08:37:02.297374810+01:00 stderr F pulp [00cdc63c3c7448c0a3b1b8195fc15371]: pulpcore.tasking.tasks:INFO:   File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py", line 68, in _execute_task
2025-02-10T08:37:02.297374810+01:00 stderr F     result = func(*args, **kwargs)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 385, in publish
2025-02-10T08:37:02.297374810+01:00 stderr F     generate_repo_metadata(
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 683, in generate_repo_metadata
2025-02-10T08:37:02.297374810+01:00 stderr F     PublishedMetadata.create_from_file(
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/publication.py", line 297, in create_from_file
2025-02-10T08:37:02.297374810+01:00 stderr F     artifact.save()
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 154, in save
2025-02-10T08:37:02.297374810+01:00 stderr F     super().save(*args, **kwargs)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
2025-02-10T08:37:02.297374810+01:00 stderr F     return func(*args, **kwds)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django_lifecycle/mixins.py", line 139, in save
2025-02-10T08:37:02.297374810+01:00 stderr F     save(*args, **kwargs)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 814, in save
2025-02-10T08:37:02.297374810+01:00 stderr F     self.save_base(
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 877, in save_base
2025-02-10T08:37:02.297374810+01:00 stderr F     updated = self._save_table(
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1020, in _save_table
2025-02-10T08:37:02.297374810+01:00 stderr F     results = self._do_insert(
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 1061, in _do_insert
2025-02-10T08:37:02.297374810+01:00 stderr F     return manager._insert(
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
2025-02-10T08:37:02.297374810+01:00 stderr F     return getattr(self.get_queryset(), name)(*args, **kwargs)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1805, in _insert
2025-02-10T08:37:02.297374810+01:00 stderr F     return query.get_compiler(using=using).execute_sql(returning_fields)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
2025-02-10T08:37:02.297374810+01:00 stderr F     for sql, params in self.as_sql():
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
2025-02-10T08:37:02.297374810+01:00 stderr F     return super(ReadonlySQLCompilerMixin, self).as_sql()
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
2025-02-10T08:37:02.297374810+01:00 stderr F     value_rows = [
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
2025-02-10T08:37:02.297374810+01:00 stderr F     [
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
2025-02-10T08:37:02.297374810+01:00 stderr F     self.prepare_value(field, self.pre_save_val(field, obj))
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
2025-02-10T08:37:02.297374810+01:00 stderr F     return field.pre_save(obj, add=True)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 89, in pre_save
2025-02-10T08:37:02.297374810+01:00 stderr F     return super().pre_save(model_instance, add)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 317, in pre_save
2025-02-10T08:37:02.297374810+01:00 stderr F     file.save(file.name, file.file, save=False)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 93, in save
2025-02-10T08:37:02.297374810+01:00 stderr F     self.name = self.storage.save(name, content, max_length=self.field.max_length)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 49, in save
2025-02-10T08:37:02.297374810+01:00 stderr F     name = self._save(name, content)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 521, in _save
2025-02-10T08:37:02.297374810+01:00 stderr F     obj.upload_fileobj(content, ExtraArgs=params, Config=self.transfer_config)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 731, in object_upload_fileobj
2025-02-10T08:37:02.297374810+01:00 stderr F     return self.meta.client.upload_fileobj(
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 642, in upload_fileobj
2025-02-10T08:37:02.297374810+01:00 stderr F     return future.result()
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
2025-02-10T08:37:02.297374810+01:00 stderr F     return self._coordinator.result()
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 264, in result
2025-02-10T08:37:02.297374810+01:00 stderr F     raise self._exception
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 135, in __call__
2025-02-10T08:37:02.297374810+01:00 stderr F     return self._execute_main(kwargs)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 158, in _execute_main
2025-02-10T08:37:02.297374810+01:00 stderr F     return_value = self._main(**kwargs)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/upload.py", line 796, in _main
2025-02-10T08:37:02.297374810+01:00 stderr F     client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
2025-02-10T08:37:02.297374810+01:00 stderr F     return self._make_api_call(operation_name, kwargs)
2025-02-10T08:37:02.297374810+01:00 stderr F 
2025-02-10T08:37:02.297374810+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
2025-02-10T08:37:02.297374810+01:00 stderr F     raise error_class(parsed_response, operation_name)
2025-02-10T08:37:02.297374810+01:00 stderr F

x9c4 · February 10, 2025, 3:42pm

Thank you for reporting this. For the sake of completeness, can you add the module invocation of your playbook here? I presume you are calling rpm_publication.

From the looks of the error message, I see that Pulp seems to have issues talking to it’s S3 storage backend. I believe this is not a squeezer issue after all. Do other operations, like upload or sync experience similar issues?

Janr · February 11, 2025, 8:34am

Sorry this was to be part of the call:

    - name: Create rpm Distribution of Repositories
      pulp.squeezer.rpm_distribution:
         name: "{{ item.name }}"
         base_path: "{{ item.base }}"
         publication: "{{ pub_result.results[d_idx].publication.pulp_href }}"
         validate_certs: false
         state: present

S3 interesting.

Are these OK:

REDIRECT_TO_OBJECT_STORAGE    = 'False'
DEFAULT_FILE_STORAGE          = 'storages.backends.s3boto3.S3Boto3Storage'

Maybe I will drop the DB and clear S3
Start fresh.

x9c4 · February 11, 2025, 9:04am

I think REDIRECT_TO_OBJECT_STORAGE should be an actual boolean the string "False" may inadvertently be cast to True… But what you actually want is a matter of your setup. If your S3 Bucket is able to serve the files directly to your clients, redirecting is highly encouraged.
I have never before seen the error message you got there. Sorry.

Janr · February 18, 2025, 9:48am

The DB was dropped and the S3 sub directory artifacts and below deleted.
After a fresh start. The error is still hard.

Would you please have a look at this as I am at wits end.
After all these years of using S3, this now happens.

https://docs.netapp.com/us-en/storagegrid-115/s3/put-object.html#resolving-conflicts

UTF-8 characters in user metadata

Could this be related?

The S3 provider is NetApp StorageGRID

The log again.

2025-02-18T07:04:29.369165419+01:00 stdout F Postgres started.
2025-02-18T07:04:29.446097455+01:00 stdout F Checking for database migrations
2025-02-18T07:04:33.756249366+01:00 stdout F Database migrated!
2025-02-18T07:04:36.762922811+01:00 stderr F pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
2025-02-18T07:04:36.851791277+01:00 stderr F pulp [None]: pulpcore.tasking.worker:INFO: New worker '1@284cb7dd83aa' discovered
2025-02-18T07:07:40.325227692+01:00 stderr F pulp [ad7a2a24e3fb49d3b3ec385a594acfc0]: pulpcore.tasking.tasks:INFO: Starting task 019517ab-58a0-74d8-9883-f154717fb8b5 in domain: default
2025-02-18T07:07:40.340812829+01:00 stderr F pulp [ad7a2a24e3fb49d3b3ec385a594acfc0]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLES12-SP4-Pool remote=remote_upstream-SLES12-SP4-Pool
2025-02-18T07:08:00.916898569+01:00 stderr F pulp [ad7a2a24e3fb49d3b3ec385a594acfc0]: pulpcore.tasking.tasks:INFO: Task[pulp_rpm.app.tasks.synchronizing.synchronize] 019517ab-58a0-74d8-9883-f154717fb8b5 failed (ClientError: An error occurred (NotImplemented) when calling the UploadPart operation: A header you provided implies functionality that is not implemented: Transfer-Encoding) in domain: default
2025-02-18T07:08:00.917319884+01:00 stderr F pulp [ad7a2a24e3fb49d3b3ec385a594acfc0]: pulpcore.tasking.tasks:INFO:   File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py", line 68, in _execute_task
2025-02-18T07:08:00.917319884+01:00 stderr F     result = func(*args, **kwargs)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 568, in synchronize
2025-02-18T07:08:00.917319884+01:00 stderr F     repo_version = dv.create() or repo.latest_version()
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
2025-02-18T07:08:00.917319884+01:00 stderr F     loop.run_until_complete(pipeline)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
2025-02-18T07:08:00.917319884+01:00 stderr F     return future.result()
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 220, in create_pipeline
2025-02-18T07:08:00.917319884+01:00 stderr F     await asyncio.gather(*futures)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 41, in __call__
2025-02-18T07:08:00.917319884+01:00 stderr F     await self.run()
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 283, in run
2025-02-18T07:08:00.917319884+01:00 stderr F     await sync_to_async(Artifact.objects.bulk_get_or_create)(
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 468, in __call__
2025-02-18T07:08:00.917319884+01:00 stderr F     ret = await asyncio.shield(exec_coro)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
2025-02-18T07:08:00.917319884+01:00 stderr F     result = self.fn(*self.args, **self.kwargs)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 522, in thread_handler
2025-02-18T07:08:00.917319884+01:00 stderr F     return func(*args, **kwargs)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 89, in bulk_get_or_create
2025-02-18T07:08:00.917319884+01:00 stderr F     return super().bulk_create(objs, batch_size=batch_size)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 87, in manager_method
2025-02-18T07:08:00.917319884+01:00 stderr F     return getattr(self.get_queryset(), name)(*args, **kwargs)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 786, in bulk_create
2025-02-18T07:08:00.917319884+01:00 stderr F     returned_columns = self._batched_insert(
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1831, in _batched_insert
2025-02-18T07:08:00.917319884+01:00 stderr F     self._insert(
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1805, in _insert
2025-02-18T07:08:00.917319884+01:00 stderr F     return query.get_compiler(using=using).execute_sql(returning_fields)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
2025-02-18T07:08:00.917319884+01:00 stderr F     for sql, params in self.as_sql():
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
2025-02-18T07:08:00.917319884+01:00 stderr F     return super(ReadonlySQLCompilerMixin, self).as_sql()
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
2025-02-18T07:08:00.917319884+01:00 stderr F     value_rows = [
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
2025-02-18T07:08:00.917319884+01:00 stderr F     [
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
2025-02-18T07:08:00.917319884+01:00 stderr F     self.prepare_value(field, self.pre_save_val(field, obj))
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
2025-02-18T07:08:00.917319884+01:00 stderr F     return field.pre_save(obj, add=True)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 89, in pre_save
2025-02-18T07:08:00.917319884+01:00 stderr F     return super().pre_save(model_instance, add)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 317, in pre_save
2025-02-18T07:08:00.917319884+01:00 stderr F     file.save(file.name, file.file, save=False)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 93, in save
2025-02-18T07:08:00.917319884+01:00 stderr F     self.name = self.storage.save(name, content, max_length=self.field.max_length)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py", line 49, in save
2025-02-18T07:08:00.917319884+01:00 stderr F     name = self._save(name, content)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/storages/backends/s3.py", line 521, in _save
2025-02-18T07:08:00.917319884+01:00 stderr F     obj.upload_fileobj(content, ExtraArgs=params, Config=self.transfer_config)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 731, in object_upload_fileobj
2025-02-18T07:08:00.917319884+01:00 stderr F     return self.meta.client.upload_fileobj(
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 642, in upload_fileobj
2025-02-18T07:08:00.917319884+01:00 stderr F     return future.result()
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 103, in result
2025-02-18T07:08:00.917319884+01:00 stderr F     return self._coordinator.result()
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/futures.py", line 264, in result
2025-02-18T07:08:00.917319884+01:00 stderr F     raise self._exception
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/botocore/httpsession.py", line 464, in send
2025-02-18T07:08:00.917319884+01:00 stderr F     urllib_response = conn.urlopen(
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 716, in urlopen
2025-02-18T07:08:00.917319884+01:00 stderr F     httplib_response = self._make_request(
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 414, in _make_request
2025-02-18T07:08:00.917319884+01:00 stderr F     conn.request_chunked(method, url, **httplib_request_kw)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 270, in request_chunked
2025-02-18T07:08:00.917319884+01:00 stderr F     for chunk in body:
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/botocore/httpchecksum.py", line 230, in __iter__
2025-02-18T07:08:00.917319884+01:00 stderr F     yield self._make_chunk()
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/botocore/httpchecksum.py", line 214, in _make_chunk
2025-02-18T07:08:00.917319884+01:00 stderr F     raw_chunk = self._raw.read(self._chunk_size)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/utils.py", line 526, in read
2025-02-18T07:08:00.917319884+01:00 stderr F     data = self._fileobj.read(amount_to_read)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/upload.py", line 91, in read
2025-02-18T07:08:00.917319884+01:00 stderr F     raise self._transfer_coordinator.exception
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 135, in __call__
2025-02-18T07:08:00.917319884+01:00 stderr F     return self._execute_main(kwargs)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/tasks.py", line 158, in _execute_main
2025-02-18T07:08:00.917319884+01:00 stderr F     return_value = self._main(**kwargs)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/s3transfer/upload.py", line 825, in _main
2025-02-18T07:08:00.917319884+01:00 stderr F     response = client.upload_part(
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
2025-02-18T07:08:00.917319884+01:00 stderr F     return self._make_api_call(operation_name, kwargs)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:00.917319884+01:00 stderr F   File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
2025-02-18T07:08:00.917319884+01:00 stderr F     raise error_class(parsed_response, operation_name)
2025-02-18T07:08:00.917319884+01:00 stderr F
2025-02-18T07:08:01.171529099+01:00 stderr F pulp [95c25607c574462bb92eae5f000a2fee]: pulpcore.tasking.tasks:INFO: Starting task 019517ab-6288-7b08-95bc-55c0c3fff10b in domain: default
2025-02-18T07:08:01.180705932+01:00 stderr F pulp [95c25607c574462bb92eae5f000a2fee]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLE-HA12-SP4-Pool remote=remote_upstream-SLE-HA12-SP4-Pool
2025-02-18T07:08:03.953911784+01:00 stderr F pulp [95c25607c574462bb92eae5f000a2fee]: pulpcore.tasking.tasks:INFO: Task[pulp_rpm.app.tasks.synchronizing.synchronize] 019517ab-6288-7b08-95bc-55c0c3fff10b failed (ClientError: An error occurred (NotImplemented) when calling the PutObject operation: A header you provided implies functionality that is not implemented: Transfer-Encoding) in domain: default
2025-02-18T07:08:03.954353905+01:00 stderr F pulp [95c25607c574462bb92eae5f000a2fee]: pulpcore.tasking.tasks:INFO:   File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py", line 68, in _execute_task
2025-02-18T07:08:03.954353905+01:00 stderr F     result = func(*args, **kwargs)
2025-02-18T07:08:03.954353905+01:00 stderr F

Thank you for the help.

x9c4 · February 18, 2025, 12:02pm

This seems to happen deep down in the boto package.
Is there any chance you can experiment with older versions of it?

ggainey · February 18, 2025, 12:59pm

We may be getting bitten by AWS’ new “Data Integrity Protections”. This NetApp article I don’t have access has the exact error you’re seeing.

Janr · February 18, 2025, 1:42pm

I have tried pulp-minimal 3.67 - it works.
I have access to:

pulp/pulp-minimal  3.63         ed9b195c5877  3 months ago   1.67 GB
boto3                                    1.35.61
botocore                                 1.35.61
s3transfer                               0.10.3

pulp/pulp-minimal  3.66         7225b397adf6  3 months ago   1.63 GB
boto3                                        1.35.54
botocore                                     1.35.54
s3transfer                                   0.10.3

pulp/pulp-minimal  3.67         ecadd3d5cea7  3 months ago   1.62 GB
boto3                                    1.35.58
botocore                                 1.35.58
s3transfer                               0.10.3

/pulp/pulp-minimal  3.68         6ea2e8a25a60  2 months ago   1.77 GB
boto3                                    1.35.77
botocore                                 1.35.77
s3transfer                               0.10.4

/pulp/pulp-minimal  3.69         73971a69fdc8  2 days ago     1.81 GB
boto3                                    1.36.21
botocore                                 1.36.21
s3transfer                               0.11.2

Thank you both for helping.
There is hope.

x9c4 · February 18, 2025, 1:50pm

A dependency of a dependency. This version of the boto3/core package is jumping around like crazy. Or is this mainly a question of when the container was built?

ggainey · February 18, 2025, 1:54pm

The AWS-client-change that Broke Things was 15-JAN, so that would fit with “3.69 Is Also Broken”. While I don’t have the AWS version-number handy, I expect “1.35.77” is It - some of what I’m seeing on The Net is ppl saying “you released a breaking change as a Z-release?!?”