POC simple reopsitory - tmp wiped afer sync

Support
#1

Problem:

Please help, what am I missing?

Setting up simple repository - pulp all-in-one container
quay.io/pulp/pulp:3.80

Sync repository from remote, on completion, tmp is cleared out.
Then the publication fails.

export RNAME=active-P-2025Q02.00-ABC-SLES12-SP4
export URL=https://ltnomad.abcgroup.net:30082/pulp/content/MR/SLES12/SP4/2025Q02.00/active-P-2025Q02.00-ABC-SLES12-SP4/


pulp rpm remote create --name remote-$RNAME \
     --url "$URL"               \
     --download-concurrency 6   \
     --tls-validation false     \
     --sock-connect-timeout 720 \
     --sock-read-timeout 720    \
     --total-timeout 7200

pulp rpm repository create --name $RNAME \
     --description "$RNAME"              \
     --metadata-signing-service 'abc_collection_signer_vault'

pulp rpm repository sync --name $RNAME --remote remote-${RNAME}

12GB is synced to /pulp_storage/tmp
As soon as the sync task completes pulp_storage/tmp is cleared.
Needless to say the publication fails

pulp rpm publication create --repository active-P-2025Q02.00-ABC-SLES12-SP4

Started background task /pulp/api/v3/tasks/01975434-e44e-70b9-a775-d19dbed2b720/
Error: Task /pulp/api/v3/tasks/01975434-e44e-70b9-a775-d19dbed2b720/ failed: '[Errno 2] No such file or directory: '/var/lib/pulp/tmp/10057@pulp_mirror/tmpql18tcdk/artifact/29/b8fd1bca28fe36e4b23c40187d20fb50b07ae68a3043924765bec5c9b9f6eb''

Sync task output.

{
  "pulp_href": "/pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/",
  "prn": "prn:core.task:019753f3-ef2e-7b14-a103-1b50a6b805a8",
  "pulp_created": "2025-06-09T09:09:44.878801Z",
  "pulp_last_updated": "2025-06-09T09:09:44.878814Z",
  "state": "completed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "cf010467f43145789422ae37a66bc1cf",
  "created_by": "/pulp/api/v3/users/1/",
  "unblocked_at": "2025-06-09T09:09:44.894082Z",
  "started_at": "2025-06-09T09:09:44.929480Z",
  "finished_at": "2025-06-09T09:15:20.453927Z",
  "error": null,
  "worker": "/pulp/api/v3/workers/019753f3-9230-7c5a-bfbc-aac9ccd2d07b/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "completed",
      "total": null,
      "done": 5,
      "suffix": null
    },
    {
      "message": "Skipping Packages",
      "code": "sync.skipped.packages",
      "state": "completed",
      "total": 0,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Parsed Packages",
      "code": "sync.parsing.packages",
      "state": "completed",
      "total": 11335,
      "done": 11335,
      "suffix": null
    },
    {
      "message": "Parsed Advisories",
      "code": "sync.parsing.advisories",
      "state": "completed",
      "total": 3880,
      "done": 3880,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "completed",
      "total": null,
      "done": 11340,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "completed",
      "total": null,
      "done": 15220,
      "suffix": null
    }
  ],
  "created_resources": [
    "/pulp/api/v3/repositories/rpm/rpm/019753f3-d7ae-7f70-b6f8-d8ce4dff81eb/versions/1/"
  ],
  "reserved_resources_record": [
    "prn:rpm.rpmrepository:019753f3-d7ae-7f70-b6f8-d8ce4dff81eb",
    "shared:prn:rpm.rpmremote:019753f3-d2cc-7259-a0c1-cab9f3ea12e4",
    "shared:prn:core.domain:285cd42b-2012-4974-80e5-1540a5609e9c"
  ]
}

Thanks so much.

Expected outcome:

sync, publication and distribution to work.

Pulpcore version:

    {
      "name": "10134@pulp_mirror",
      "last_heartbeat": "2025-06-09T10:28:59.735775Z",
      "versions": {
        "deb": "3.5.2",
        "gem": "0.7.1",
        "npm": "0.3.3",
        "rpm": "3.30.1",
        "core": "3.80.0",
        "file": "3.80.0",
        "maven": "0.10.1",
        "ostree": "2.4.8",
        "python": "3.15.0",
        "ansible": "0.26.0",
        "certguard": "3.80.0",
        "container": "2.25.1"
      }

Pulp plugins installed and their versions:

pip                25.1.1
pulp-cli           0.33.0
pulp-glue          0.33.0

Operating system - distribution and version:

Pulp container hosted on SLES15 SP5

Other relevant data:

Container log

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
fix-attrs: info: applying /etc/fix-attrs.d/postgres
/package/admin/s6-overlay-3.1.6.2/etc/s6-rc/scripts/fix-attrs: line 16: 49 Segmentation fault (core dumped) /package/admin/s6-overlay-3.1.6.2/libexec/fix-attrs < “$1”
fix-attrs: warning: fix-attrs is deprecated, please fix volume permissions in your container manager instead
fix-attrs: warning: some fix files failed to apply
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service postgres-init: starting
s6-rc: info: service fix-pulp-perms: starting
s6-rc: info: service db-fields-key-create: starting
s6-rc: info: service redis: starting
Calling /etc/init/db-fields-key-create
Calling /etc/init/postgres-init
s6-rc: info: service redis successfully started
Calling /etc/init/fix-pulp-perms
[oneshot] db-fields-key-create: creating database fields key
s6-rc: info: service db-fields-key-create successfully started
s6-rc: info: service certs: starting
Calling /etc/init/certs
[oneshot] certs: adding webserver certificate to the certificate store
Checking /var/lib/pulp/media permissions
Checking /var/lib/pulp/scripts permissions
Checking /var/lib/pulp/tmp permissions
[oneshot] postgres-init: initdb -E UTF8 --locale=C.UTF-8 --pgdata /var/lib/pgsql/data
s6-rc: info: service fix-pulp-perms successfully started
The files belonging to this database system will be owned by user “postgres”.
This user must also own the server process.

The database cluster will be initialized with locale “C.UTF-8”.
The default text search configuration will be set to “english”.

Data page checksums are disabled.

creating directory /var/lib/pgsql/data 
 ok
creating subdirectories 
 ok
selecting dynamic shared memory implementation 
 posix
selecting default max_connections 
 100
selecting default shared_buffers 
 128MB
selecting default time zone 
 Europe/Berlin
creating configuration files 
 ok
running bootstrap script 
 ok
performing post-bootstrap initialization 
 ok
syncing data to disk 
 ok

initdb: warning: enabling “trust” authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
–auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

pg_ctl -D /var/lib/pgsql/data -l logfile start

s6-rc: info: service postgres-init successfully started
s6-rc: info: service postgresql: starting
s6-rc: info: service postgresql successfully started
s6-rc: info: service postgres-prepare: starting
Calling /etc/init/postgres-prepare
2025-06-09 11:08:11.130 CEST [9759] LOG: redirecting log output to logging collector process
2025-06-09 11:08:11.130 CEST [9759] HINT: Future log output will appear in directory “log”.
[oneshot] certs: finished adding webserver certificate to the certificate store
s6-rc: info: service certs successfully started
[oneshot] postgres-prepare: createuser pulp
[oneshot] postgres-prepare: createdb --encoding=utf-8 --locale=en_US.UTF-8 -T template0 -O pulp pulp
[oneshot] postgres-prepare: running Pulp migrations
Operations to perform:
Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, gem, maven, npm, ostree, python, rpm, sessions
Running migrations:
Applying contenttypes.0001_initial
 OK
Applying contenttypes.0002_remove_content_type_name
 OK
Applying auth.0001_initial
 OK
Applying auth.0002_alter_permission_name_max_length
 OK
Applying auth.0003_alter_user_email_max_length
 OK
Applying auth.0004_alter_user_username_opts
 OK
Applying auth.0005_alter_user_last_login_null
 OK
Applying auth.0006_require_contenttypes_0002
 OK
Applying auth.0007_alter_validators_add_error_messages
 OK
Applying auth.0008_alter_user_username_max_length
 OK
Applying auth.0009_alter_user_last_name_max_length
 OK
Applying auth.0010_alter_group_name_max_length
 OK
Applying auth.0011_update_proxy_permissions
 OK
Applying auth.0012_alter_user_first_name_max_length
 OK
Applying core.0001_squashed_0090_char_to_text_field
 OK
Applying core.0091_systemid
 OK
Applying core.0092_alter_upload_options
 OK
Applying core.0093_add_info_field_repositoryversion
 OK
Applying core.0094_protect_repository_content
 OK
Applying core.0095_artifactdistribution
 OK
Applying core.0096_alter_task_logging_cid
 OK
Applying core.0097_remove_telemetry_task_schedule
 OK
Applying core.0098_pulp_labels
 OK
Applying core.0099_versions_field
 OK
.
.
.
Applying container.0028_add_role_manage_permissions
 OK
Applying container.0029_remove_blob_media_type
 OK
Applying container.0030_enforce_tagged_manifest_reference
 OK
Applying container.0031_replace_charf_with_textf
 OK
Applying container.0032_upload_artifact
 OK
Applying container.0033_raise_warning_for_repair
 OK
Applying container.0034_translate_signed_schema
 OK
Applying container.0035_alter_blob_content_ptr_and_more
 OK
Applying container.0036_containerpushrepository_pending_blobs_manifests
 OK
Applying container.0037_create_pull_through_cache_models
 OK
/usr/local/lib/python3.9/site-packages/pulp_container/app/migrations/0038_add_manifest_metadata_fields.py:8: UserWarning: Run ‘pulpcore-manager container-handle-image-data’ to initialize and expose metadata (i.e., annotations and labels) for all manifests.
warnings.warn(
Applying container.0038_add_manifest_metadata_fields
 OK
/usr/local/lib/python3.9/site-packages/pulp_container/app/migrations/0039_manifest_data.py:8: UserWarning: Run ‘pulpcore-manager container-handle-image-data’ to move the manifests’ data from artifacts to the new ‘data’ database field.
warnings.warn(
Applying container.0039_manifest_data
 OK
Applying container.0040_add_remote_repo_filter
 OK
Applying container.0041_add_pull_through_pull_permissions
 OK
Applying container.0042_add_manifest_nature_field
 OK
/usr/local/lib/python3.9/site-packages/pulp_container/app/migrations/0043_add_os_arch_image_size_manifest_fields.py:7: UserWarning: Run ‘pulpcore-manager container-handle-image-data’ to update the manifests’ os, architecture, and compressed_image_size fields.
warnings.warn(
Applying container.0043_add_os_arch_image_size_manifest_fields
 OK
Applying container.0044_add_domain
 OK
Applying container.0045_alter_manifest_compressed_image_size
 OK
Applying core.0126_remoteartifact_failed_at
 OK
Applying core.0127_remove_upstreampulp_pulp_label_select
 OK
Applying core.0128_domain_pulp_labels
 OK
Applying core.0129_content_pulp_labels
 OK
Applying core.0130_upstreampulp_policy
 OK
Applying core.0132_alter_content_options
 OK
Applying core.0131_distribution_checkpoint_publication_checkpoint
 OK
Applying deb.0001_initial
 OK
Applying deb.0002_auto_20190905_1000
 OK
Applying deb.0003_rename_package_field
 OK
Applying deb.0004_auto_20191014_1722
 OK
Applying deb.0005_rename_release_file
 OK
Applying deb.0006_debrepository
 OK
Applying deb.0007_create_metadata_models
 OK
Applying deb.0008_debremote_gpgkey
 OK
Applying deb.0009_apt_release_signing_service
 OK
Applying deb.0010_debpublication_signing_service
 OK
Applying deb.0011_rename_models_to_apt
 OK
Applying deb.0012_auto_20200803_1337
 OK
Applying deb.0013_aptremote_ignore_missing_package_indices
 OK
Applying deb.0014_swap_distribution_model
 OK
Applying deb.0015_add_custom_fields_to_packages
 OK
Applying deb.0016_switch_to_newer_jsonfield_model
 OK
Applying deb.0017_allow_longer_string_lists
 OK
Applying deb.0018_textfield_conversion
 OK
Applying deb.0019_immutable_metadata_constraints
 OK
Applying deb.0020_remove_fk_relations_in_packageindex_and_installerfileindex
 OK
Applying deb.0021_remove_release_from_structure_types
 OK
Applying deb.0022_alter_aptdistribution_distribution_ptr_and_more
 OK
Applying deb.0023_add_default_signing_services
 OK
Applying deb.0024_add_release_fields
 OK
pulp [None]: pulp_deb.app.migrations.0025_merge_colliding_structure_content:INFO: 2025-06-09 09:08:35.163663: Starting data migration!
pulp [None]: pulp_deb.app.migrations.0025_merge_colliding_structure_content:INFO: 2025-06-09 09:08:35.165197: Data migration completed!

Applying deb.0025_merge_colliding_structure_content


OK
Applying deb.0026_aptrepository_publish_upstream_release_fields
 OK
Applying deb.0027_alter_aptpublication_structured
 OK
Applying deb.0028_sourcepackage_sourcepackagereleasecomponent_and_more
 OK
Applying deb.0029_distributedpublication
 OK
Applying deb.0030_rbac_permissions
 OK
Applying file.0001_initial_squashed_0016_add_domain
 OK
Applying file.0017_alter_filealternatecontentsource_alternatecontentsource_ptr_and_more
 OK
Applying gem.0001_initial_squashed_0011_alter_gemcontent_platform
 OK
Applying gem.0012_alter_gemdistribution_options_and_more
 OK
Applying maven.0001_initial
 OK
Applying maven.0002_maven_related_names
 OK
Applying maven.0003_mavenrepository
 OK
Applying maven.0004_swap_distribution_models
 OK
Applying maven.0005_mavenmetadata
 OK
Applying maven.0006_alter_mavenartifact_content_ptr_and_more
 OK
Applying maven.0007_alter_mavenartifact_unique_together_and_more
 OK
Applying npm.0001_initial
 OK
Applying npm.0002_swap_distribution_model
 OK
Applying npm.0003_alter_npmdistribution_distribution_ptr_and_more
 OK
Applying npm.0004_alter_package_unique_together_package__pulp_domain_and_more
 OK
Applying ostree.0001_initial
 OK
Applying ostree.0002_add_relative_path_uniqueness
 OK
Applying ostree.0003_create_many_to_many_objs_commits
 OK
Applying ostree.0004_add_include_exclude_refs
 OK
Applying ostree.0005_add_static_delta_support
 OK
.
.
.
.
.
. Applying rpm.0052_modulemd_digest
 OK
Applying rpm.0053_rpmdistribution_generate_repo_config
 OK
Applying rpm.0054_remove_gpg_fields
 OK
Applying rpm.0055_add_repo_config_field
 OK
Applying rpm.0056_remove_rpmpublication_sqlite_metadata_and_more
 OK
Applying rpm.0057_rpmpublication_checksum_type_and_more
 OK
Applying rpm.0058_alter_addon_repository_alter_variant_repository
 OK
Applying rpm.0059_rpmpublication_compression_type_and_more
 OK
Applying rpm.0060_rpmpublication_compression_type_empty
 OK
Applying rpm.0061_fix_modulemd_defaults_digest
 OK
Applying rpm.0062_rpmpackagesigningservice_and_more
 OK
Applying rpm.0063_rpmpublication_layout_rpmrepository_layout
 OK
Applying rpm.0064_remove_rpmrepository_original_checksum_types_and_more
 OK
Applying sessions.0001_initial
 OK
Access policy for artifacts created.
Access policy for contentguards/core/composite created.
.
.
.
.
.
.Access policy for publications/file/file created.
Access policy for remotes/file/file created.
Access policy for repositories/file/file/versions created.
Access policy for repositories/file/file created.
[oneshot] postgres-prepare: Setting Pulp admin password
Successfully set password for “admin” user.
s6-rc: info: service postgres-prepare successfully started
s6-rc: info: service add-workers: starting
s6-rc: info: service pulpcore-worker@1: starting
s6-rc: info: service pulpcore-content: starting
s6-rc: info: service pulpcore-api: starting
s6-rc: info: service pulpcore-worker@1 successfully started
s6-rc: info: service pulpcore-content successfully started
s6-rc: info: service pulpcore-api successfully started
s6-rc: info: service nginx: starting
Checking for database migrations
Calling /etc/init/add-workers
s6-rc: info: service nginx successfully started
Checking for database migrations
Checking for database migrations
Calling /etc/init/nginx
[oneshot] add-workers: Adding workers 2 through 2
[oneshot] add-workers: Workers will be added shortly after the boot finishes
s6-rc: info: service add-workers successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
Calling /etc/init/postgres-init
Calling /etc/init/db-fields-key-create
Calling /etc/init/fix-pulp-perms
[oneshot] db-fields-key-create: creating database fields key
Calling /etc/init/certs
Checking /var/lib/pulp/media permissions
Checking /var/lib/pulp/scripts permissions
Checking /var/lib/pulp/tmp permissions
Calling /etc/init/postgres-prepare
2025-06-09 11:09:12.292 CEST [9989] LOG: redirecting log output to logging collector process
2025-06-09 11:09:12.292 CEST [9989] HINT: Future log output will appear in directory “log”.
ALTER DATABASE
[oneshot] postgres-prepare: running Pulp migrations
Database migrated!
ready
Database migrated!
ready
Database migrated!
ready
Operations to perform:
Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, gem, maven, npm, ostree, python, rpm, sessions
Running migrations:
No migrations to apply.
Calling /etc/init/add-workers
Already migrated!
Already migrated!
Already migrated!
Calling /etc/init/nginx
Calling /etc/init/pulpcore-content
Calling /etc/init/pulpcore-worker
/usr/local/bin/pulpcore-content
Calling /etc/init/pulpcore-api
/usr/local/bin/pulpcore-api
[oneshot] add-workers: Adding workers 2 through 2
[oneshot] add-workers: Workers will be added shortly after the boot finishes
s6-rc: fatal: unable to take locks: Resource busy
Already migrated!
Calling /etc/init/pulpcore-worker
[2025-06-09 11:09:19 +0200] [10061] [INFO] Starting gunicorn 23.0.0
[2025-06-09 11:09:19 +0200] [10061] [INFO] Listening at: http://[::]:24817 (10061)
[2025-06-09 11:09:19 +0200] [10061] [INFO] Using worker: pulpcore.app.entrypoint.PulpApiWorker
[2025-06-09 11:09:19 +0200] [10110] [INFO] Booting worker with pid: 10110
[2025-06-09 11:09:19 +0200] [10111] [INFO] Booting worker with pid: 10111
pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
pulp [None]: pulpcore.tasking.worker:INFO: New worker ‘10057@pulp_mirror’ discovered
pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
pulp [None]: pulpcore.tasking.worker:INFO: New worker ‘10105@pulp_mirror’ discovered
[2025-06-09 09:09:21 +0000] [10060] [INFO] Starting gunicorn 23.0.0
[2025-06-09 09:09:21 +0000] [10060] [INFO] Listening at: http://[::]:24816 (10060)
[2025-06-09 09:09:21 +0000] [10060] [INFO] Using worker: aiohttp.GunicornWebWorker
[2025-06-09 09:09:21 +0000] [10134] [INFO] Booting worker with pid: 10134
[2025-06-09 09:09:21 +0000] [10137] [INFO] Booting worker with pid: 10137
pulp [f8bd5f92c3234d0a87c2a798349a1167]: pulpcore.tasking._util:INFO: Dispatched scheduled task pulpcore.app.tasks.analytics.post_analytics as task id 019753f3-b929-7c4f-8235-b31788661e13
pulp [f8bd5f92c3234d0a87c2a798349a1167]: pulpcore.tasking.tasks:INFO: Starting task id: 019753f3-b929-7c4f-8235-b31788661e13 in domain: default, task_type: pulpcore.app.tasks.analytics.post_analytics, immediate: False, deferred: True
pulp [f8bd5f92c3234d0a87c2a798349a1167]: pulpcore.app.tasks.analytics:ERROR: Error sending analytics to https://analytics.pulpproject.org/: ClientConnectorDNSError(ConnectionKey(host=‘analytics.pulpproject.org’, port=443, is_ssl=True, ssl=True, proxy=None, proxy_auth=None, proxy_headers_hash=None), OSError(None, ‘Domain name not found’))
pulp [f8bd5f92c3234d0a87c2a798349a1167]: pulpcore.tasking.tasks:INFO: Task completed 019753f3-b929-7c4f-8235-b31788661e13 in domain: default, task_type: pulpcore.app.tasks.analytics.post_analytics, immediate: False, deferred: True, execution_time: 25019 ÎŒs
(‘pulp [b20996324de34b8ba8a698b4ec6edfa0]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:37 +0000] “POST /pulp/api/v3/remotes/rpm/rpm/ HTTP/1.0” 201 933 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [784abdecc9ef4c82a36979ca63699658]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:38 +0000] “GET /pulp/api/v3/signing-services/?name=bmw_collection_signer_vault&offset=0&limit=1 HTTP/1.0” 200 3882 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [784abdecc9ef4c82a36979ca63699658]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:38 +0000] “GET /pulp/api/v3/remotes/rpm/rpm/?name=remote-active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 985 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [784abdecc9ef4c82a36979ca63699658]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:38 +0000] “POST /pulp/api/v3/repositories/rpm/rpm/ HTTP/1.0” 201 1041 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [0f0f6c1f977c4ac6a62e9eff0cadefeb]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:39 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/?name=active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 1093 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:44 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/?name=active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 1093 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:44 +0000] “POST /pulp/api/v3/repositories/rpm/rpm/019753f3-d7ae-7f70-b6f8-d8ce4dff81eb/sync/ HTTP/1.0” 202 67 “-” “Pulp-CLI/0.33.0”’,)
pulp [cf010467f43145789422ae37a66bc1cf]: pulpcore.tasking.tasks:INFO: Starting task id: 019753f3-ef2e-7b14-a103-1b50a6b805a8 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True
pulp [cf010467f43145789422ae37a66bc1cf]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=active-P-2025Q02.00-BMW-SLES12-SP4 remote=remote-active-P-2025Q02.00-BMW-SLES12-SP4
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:45 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 890 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:46 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1511 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:47 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1511 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:48 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1512 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:49 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1513 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:50 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1513 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:51 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1513 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:53 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1513 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:09:54 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1513 “-” “Pulp-CLI/0.33.0”’,)
.
.
.repeats removed
.
.
.(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:15:15 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1649 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:15:17 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1649 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:15:18 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1651 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:15:19 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1653 “-” “Pulp-CLI/0.33.0”’,)
pulp [cf010467f43145789422ae37a66bc1cf]: pulpcore.tasking.tasks:INFO: Task completed 019753f3-ef2e-7b14-a103-1b50a6b805a8 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 335524447 ÎŒs
(‘pulp [cf010467f43145789422ae37a66bc1cf]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:15:20 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1749 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [f4d2906841104662b4ec7f5853f1a23d]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:15:25 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1749 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [9655c1887bd044ffbe57c0d3056842d5]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:24:24 +0000] “GET /pulp/api/v3/tasks/019753f3-ef2e-7b14-a103-1b50a6b805a8/ HTTP/1.0” 200 1749 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [f4012fd44e91480eb88dff8240ae4120]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:28:45 +0000] “GET /pulp/api/v3/remotes/rpm/rpm/?offset=0&limit=25 HTTP/1.0” 200 985 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [f034f12d925146a29eee755f45a55037]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:32:08 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/?name=remote-active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 52 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [49dd0936c4334aa48162799522a9cd9b]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:32:20 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/?name=active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 1093 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [49dd0936c4334aa48162799522a9cd9b]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:32:20 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/019753f3-d7ae-7f70-b6f8-d8ce4dff81eb/versions/1/ HTTP/1.0” 200 1552 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [fcbb1a75e00144b99390c86a92e878e5]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:57:07 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/?name=active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 1093 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [fcbb1a75e00144b99390c86a92e878e5]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:57:07 +0000] “GET /pulp/api/v3/remotes/rpm/rpm/?name=remote-active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 985 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [fcbb1a75e00144b99390c86a92e878e5]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:57:07 +0000] “POST /pulp/api/v3/repositories/rpm/rpm/019753f3-d7ae-7f70-b6f8-d8ce4dff81eb/sync/ HTTP/1.0” 202 67 “-” “Pulp-CLI/0.33.0”’,)
pulp [fcbb1a75e00144b99390c86a92e878e5]: pulpcore.tasking.tasks:INFO: Starting task id: 0197541f-4f1c-7d7f-90d0-89fa63ae336f in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True
pulp [fcbb1a75e00144b99390c86a92e878e5]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=active-P-2025Q02.00-BMW-SLES12-SP4 remote=remote-active-P-2025Q02.00-BMW-SLES12-SP4
(‘pulp [fcbb1a75e00144b99390c86a92e878e5]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:57:07 +0000] “GET /pulp/api/v3/tasks/0197541f-4f1c-7d7f-90d0-89fa63ae336f/ HTTP/1.0” 200 890 “-” “Pulp-CLI/0.33.0”’,)
pulp [fcbb1a75e00144b99390c86a92e878e5]: pulpcore.tasking.tasks:INFO: Task completed 0197541f-4f1c-7d7f-90d0-89fa63ae336f in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 437722 ÎŒs
(‘pulp [fcbb1a75e00144b99390c86a92e878e5]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:57:08 +0000] “GET /pulp/api/v3/tasks/0197541f-4f1c-7d7f-90d0-89fa63ae336f/ HTTP/1.0” 200 1054 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [61dfd159ff94404fbd1ea838400a4177]: ::ffff:127.0.0.1 - admin [09/Jun/2025:09:58:00 +0000] “GET /pulp/api/v3/tasks/0197541f-4f1c-7d7f-90d0-89fa63ae336f/ HTTP/1.0” 200 1054 “-” “Pulp-CLI/0.33.0”’,)
::ffff:127.0.0.1 [09/Jun/2025:09:58:32 +0000] “GET /pulp/content/MR/SLES12/SP4/2025Q02.00/ HTTP/1.0” 404 174 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36”
::ffff:127.0.0.1 [09/Jun/2025:09:58:43 +0000] “GET /pulp/content/MR/SLES12/SP4/ HTTP/1.0” 404 174 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36”
(‘pulp [2431436bfd5041368649d21cb9415163]: ::ffff:127.0.0.1 - admin [09/Jun/2025:10:09:57 +0000] “GET /pulp/api/v3/publications/rpm/rpm/?offset=0&limit=25 HTTP/1.0” 200 52 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [9dd7fae08e974b12a7d951b5b5f7145f]: ::ffff:127.0.0.1 - admin [09/Jun/2025:10:20:32 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/?name=remote-active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 52 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [b05fd01604284a65b9a6b93dad80ceaa]: ::ffff:127.0.0.1 - admin [09/Jun/2025:10:20:41 +0000] “GET /pulp/api/v3/repositories/rpm/rpm/?name=active-P-2025Q02.00-BMW-SLES12-SP4&offset=0&limit=1 HTTP/1.0” 200 1093 “-” “Pulp-CLI/0.33.0”’,)
(‘pulp [b05fd01604284a65b9a6b93dad80ceaa]: ::ffff:127.0.0.1 - admin [09/Jun/2025:10:20:41 +0000] “POST /pulp/api/v3/publications/rpm/rpm/ HTTP/1.0” 202 67 “-” “Pulp-CLI/0.33.0”’,)
pulp [b05fd01604284a65b9a6b93dad80ceaa]: pulpcore.tasking.tasks:INFO: Starting task id: 01975434-e44e-70b9-a775-d19dbed2b720 in domain: default, task_type: pulp_rpm.app.tasks.publishing.publish, immediate: False, deferred: True
pulp [b05fd01604284a65b9a6b93dad80ceaa]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=active-P-2025Q02.00-BMW-SLES12-SP4, version=1
pulp [b05fd01604284a65b9a6b93dad80ceaa]: pulpcore.tasking.tasks:INFO: Task[pulp_rpm.app.tasks.publishing.publish] 01975434-e44e-70b9-a775-d19dbed2b720 failed (FileNotFoundError: [Errno 2] No such file or directory: ‘/var/lib/pulp/tmp/10057@pulp_mirror/tmpql18tcdk/artifact/29/b8fd1bca28fe36e4b23c40187d20fb50b07ae68a3043924765bec5c9b9f6eb’) in domain: default
pulp [b05fd01604284a65b9a6b93dad80ceaa]: pulpcore.tasking.tasks:INFO: File “/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py”, line 94, in _execute_task
func(*args, **kwargs)

File “/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py”, line 385, in publish
publication_data.populate()

File “/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py”, line 281, in populate
self.repomdrecords = self.prepare_metadata_files(main_content)

File “/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py”, line 106, in prepare_metadata_files
current_file = content_artifact.artifact.file.file

File “/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py”, line 48, in _get_file
self._file = self.storage.open(self.name, “rb”)

File “/usr/local/lib/python3.9/site-packages/django/core/files/storage/base.py”, line 22, in open
return self._open(name, mode)

File “/usr/local/lib/python3.9/site-packages/django/core/files/storage/filesystem.py”, line 68, in _open
return File(open(self.path(name), mode))

(‘pulp [b05fd01604284a65b9a6b93dad80ceaa]: ::ffff:127.0.0.1 - admin [09/Jun/2025:10:20:42 +0000] “GET /pulp/api/v3/tasks/01975434-e44e-70b9-a775-d19dbed2b720/ HTTP/1.0” 200 2118 “-” “Pulp-CLI/0.33.0”’,)

#2

Hmmm. 3.30.1 includes a change to improve publish-performance - you may have found an edge-case for us (alas). @dralley , take a look at here - let’s see if we can figure out what’s going on today.

#3

Much appreciated.

#4

I did a quick experiment with centos-stream-10 and and was successful - altho I did get a “harmless” error along the way - not sure if it’s at all related. I’m dropping it here as part of information-gathering:

pulp-1   | pulp [f2dfdcc328b64868a673505432cbc175]: pulpcore.tasking.tasks:INFO: Starting task id: 019754ed-0505-7445-81c1-83f4e9988d6e in domain: default, task_type: pulp_rpm.app.tasks.publishing.publish, immediate: False, deferred: True
pulp-1   | pulp [f2dfdcc328b64868a673505432cbc175]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=centos, version=1
pulp-1   | ('pulp [f2dfdcc328b64868a673505432cbc175]: ::ffff:127.0.0.1 - admin [09/Jun/2025:13:41:49 +0000] "GET /pulp/default/api/v3/tasks/019754ed-0505-7445-81c1-83f4e9988d6e/ HTTP/1.0" 200 931 "-" "Pulp-CLI/0.34.0.dev"',)
pulp-1   | Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7fca0c8cf8b0>
pulp-1   | Traceback (most recent call last):
pulp-1   |   File "/usr/lib64/python3.9/tempfile.py", line 461, in __del__
pulp-1   |     self.close()
pulp-1   |   File "/usr/lib64/python3.9/tempfile.py", line 457, in close
pulp-1   |     unlink(self.name)
pulp-1   | FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/tmp/1104@f88fe8280dc5/tmpolkkc2dl/tmp25xwkwpe'

Publish succeeded just fine in this instance, against remote “Index of /10-stream/BaseOS/x86_64/os”

One question - you have a metadata-signing-service. Is there any possibility we can see that code? Alternately/in-addition - can you try a sync-and-publish that does not use that? Trying to minimize variables here.

#5

Temporary storage is independent for each task, so the working directory for a sync task being deleted should not impact a subsequent publish task at all. Unless somehow the entire /tmp/ directory is being deleted in which case the creation of the next temporary working directory might fail I suppose. But that doesn’t look like what’s happening here?

The error is

No such file or directory: ‘/var/lib/pulp/tmp/10057@pulp_mirror/tmpql18tcdk/artifact/29/b8fd1bca28fe36e4b23c40187d20fb50b07ae68a3043924765bec5c9b9f6eb

This kind of looks to me like the temporary location and the permanent location got squished together and we’re passing the wrong path somehow. Which is maybe true with Grant’s error as well even though his is different.

#6

FWIW I also don’t really see anything in the recent publish patch which would impact this at all.

#7

Hi, I will test tomorrow without signing.

tmp contains only two empty directories.

l /global/podmandata/pulp-mirror/pulp_storage/tmp
total 0
drwxr-sr-x 4 700 700 56 Jun  9 15:35  ./
drwxr-sr-x 7 700 700 94 Jun  4 11:13  ../
drwx--S--- 2 700 700  6 Jun  9 15:18 '10067@pulp_mirror'/
drwx--S--- 2 700 700  6 Jun  9 15:19 '10114@pulp_mirror'/

If another sync method is used mirror_complete the distribution is created and repomd.xml signed.
But tmp and artifacts empty - so downloading from the produced dist url fails with a 500 as the directories are empty.

#8

A repeat build without the “signer”.
Removing the signer had no affect.

export RNAME=active-P-2025Q02.00-ABC-SLES12-SP4
export URL=https://ltnomad.abcgroup.net:30082/pulp/content/MR/SLES12/SP4/2025Q02.00/active-P-2025Q02.00-ABC-SLES12-SP4/

pulp rpm remote create --name remote-$RNAME \
     --url "$URL"               \
     --download-concurrency 6   \
     --tls-validation false     \
     --sock-connect-timeout 720 \
     --sock-read-timeout 720    \
     --total-timeout 7200

pulp rpm repository create --name $RNAME \
     --description "$RNAME"              \
     --remote  "remote-$RNAME"

export REPO_HREF=$(pulp rpm repository show --name $RNAME|jq -r '.pulp_href')
echo REPO_HREF = $REPO_HREF


echo repo
echo ======================================

Start of build:
Create remote:

{
  "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/0197584b-2cc5-7cc4-9813-366ede26314a/",
  "prn": "prn:rpm.rpmremote:0197584b-2cc5-7cc4-9813-366ede26314a",
  "pulp_created": "2025-06-10T05:23:31.142529Z",
  "pulp_last_updated": "2025-06-10T05:23:31.142556Z",
  "name": "remote-active-P-2025Q02.00-BMW-SLES12-SP4",
  "url": "https://ltnomadcli01.bmwgroup.net:30082/pulp/content/MR/SLES12/SP4/2025Q02.00/active-P-2025Q02.00-BMW-SLES12-SP4/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": false,
  "proxy_url": null,
  "pulp_labels": {},
  "download_concurrency": 6,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": 7200.0,
  "connect_timeout": null,
  "sock_connect_timeout": 720.0,
  "sock_read_timeout": 720.0,
  "headers": null,
  "rate_limit": null,
  "hidden_fields": [
    {
      "name": "client_key",
      "is_set": false
    },
    {
      "name": "proxy_username",
      "is_set": false
    },
    {
      "name": "proxy_password",
      "is_set": false
    },
    {
      "name": "username",
      "is_set": false
    },
    {
      "name": "password",
      "is_set": false
    }
  ],
  "sles_auth_token": null
}
{
  "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/0197584b-30b7-782f-a5d6-bac21b860e1d/",
  "prn": "prn:rpm.rpmrepository:0197584b-30b7-782f-a5d6-bac21b860e1d",
  "pulp_created": "2025-06-10T05:23:32.152411Z",
  "pulp_last_updated": "2025-06-10T05:23:32.161503Z",
  "versions_href": "/pulp/api/v3/repositories/rpm/rpm/0197584b-30b7-782f-a5d6-bac21b860e1d/versions/",
  "pulp_labels": {},
  "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/0197584b-30b7-782f-a5d6-bac21b860e1d/versions/0/",
  "name": "active-P-2025Q02.00-BMW-SLES12-SP4",
  "description": "active-P-2025Q02.00-BMW-SLES12-SP4",
  "retain_repo_versions": null,
  "remote": "/pulp/api/v3/remotes/rpm/rpm/0197584b-2cc5-7cc4-9813-366ede26314a/",
  "autopublish": false,
  "metadata_signing_service": null,
  "package_signing_service": null,
  "package_signing_fingerprint": "",
  "retain_package_versions": 0,
  "checksum_type": null,
  "metadata_checksum_type": null,
  "package_checksum_type": null,
  "sqlite_metadata": false,
  "repo_config": {},
  "compression_type": null,
  "layout": null
}
REPO_HREF = /pulp/api/v3/repositories/rpm/rpm/0197584b-30b7-782f-a5d6-bac21b860e1d/
repo - end
======================================
Start sync
Started background task /pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/
.........................................................................................................................................................................................................................................................................................................................................................................................................................Done.


du -hs /global/podmandata/pulp-mirror/pulp_storage/*

0       /global/podmandata/pulp-mirror/pulp_storage/media
8.0K    /global/podmandata/pulp-mirror/pulp_storage/scripts
du: cannot access '/global/podmandata/pulp-mirror/pulp_storage/tmp/10063@pulp_mirror/tmp8sjzw9_a/artifact/f9/029e233919e1bdae2560be191a7af4264fbb3509cfaa587b317f4cedf555dd': No such file or directory
du: cannot access '/global/podmandata/pulp-mirror/pulp_storage/tmp/10063@pulp_mirror/tmp8sjzw9_a/artifact/f9/06e9f11de04f04e4e901a10c1292cb69960c77d7260c02c6ad880ef56d4fff': No such file or directory
2.0G    /global/podmandata/pulp-mirror/pulp_storage/tmp
sync - end
=====================================

Seconds before disaster

pulp task show  --uuid 0197584b-48f7-7ae8-9e80-5d4f3e27ab4a ; du -sh /global/podmandata/pulp-mirror/pulp_storage/tmp/* /global/podmandata/pulp-mirror/pulp_storage/media/
{
  "pulp_href": "/pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/",
  "prn": "prn:core.task:0197584b-48f7-7ae8-9e80-5d4f3e27ab4a",
  "pulp_created": "2025-06-10T05:23:38.360391Z",
  "pulp_last_updated": "2025-06-10T05:23:38.360404Z",
  "state": "running",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "d64bd8ceab924eca9d08092552ffe885",
  "created_by": "/pulp/api/v3/users/1/",
  "unblocked_at": "2025-06-10T05:23:38.375759Z",
  "started_at": "2025-06-10T05:23:38.412487Z",
  "finished_at": null,
  "error": null,
  "worker": "/pulp/api/v3/workers/01975849-4467-7fbd-8bdd-1d60a18625a6/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "completed",
      "total": null,
      "done": 5,
      "suffix": null
    },
    {
      "message": "Skipping Packages",
      "code": "sync.skipped.packages",
      "state": "completed",
      "total": 0,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Parsed Packages",
      "code": "sync.parsing.packages",
      "state": "completed",
      "total": 11335,
      "done": 11335,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "running",
      "total": null,
      "done": 11000,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "running",
      "total": null,
      "done": 11196,
      "suffix": null
    },
    {
      "message": "Parsed Advisories",
      "code": "sync.parsing.advisories",
      "state": "running",
      "total": 3880,
      "done": 1691,
      "suffix": null
    }
  ],
  "created_resources": [
    "<unavailable>"
  ],
  "reserved_resources_record": [
    "prn:rpm.rpmrepository:0197584b-30b7-782f-a5d6-bac21b860e1d",
    "shared:prn:rpm.rpmremote:0197584b-2cc5-7cc4-9813-366ede26314a",
    "shared:prn:core.domain:6a62c031-6b33-414b-a60e-ddd3675aa73d"
  ]
}

12G     /global/podmandata/pulp-mirror/pulp_storage/tmp/10063@pulp_mirror
0       /global/podmandata/pulp-mirror/pulp_storage/tmp/10111@pulp_mirror
0       /global/podmandata/pulp-mirror/pulp_storage/media/

pulp task show  --uuid 0197584b-48f7-7ae8-9e80-5d4f3e27ab4a ; du -sh /global/podmandata/pulp-mirror/pulp_storage/tmp/* /global/podmandata/pulp-mirror/pulp_storage/media/
{
  "pulp_href": "/pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/",
  "prn": "prn:core.task:0197584b-48f7-7ae8-9e80-5d4f3e27ab4a",
  "pulp_created": "2025-06-10T05:23:38.360391Z",
  "pulp_last_updated": "2025-06-10T05:23:38.360404Z",
  "state": "completed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "d64bd8ceab924eca9d08092552ffe885",
  "created_by": "/pulp/api/v3/users/1/",
  "unblocked_at": "2025-06-10T05:23:38.375759Z",
  "started_at": "2025-06-10T05:23:38.412487Z",
  "finished_at": "2025-06-10T05:31:27.685569Z",
  "error": null,
  "worker": "/pulp/api/v3/workers/01975849-4467-7fbd-8bdd-1d60a18625a6/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "completed",
      "total": null,
      "done": 5,
      "suffix": null
    },
    {
      "message": "Skipping Packages",
      "code": "sync.skipped.packages",
      "state": "completed",
      "total": 0,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Parsed Advisories",
      "code": "sync.parsing.advisories",
      "state": "completed",
      "total": 3880,
      "done": 3880,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "completed",
      "total": null,
      "done": 11340,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "completed",
      "total": null,
      "done": 15220,
      "suffix": null
    },
    {
      "message": "Parsed Packages",
      "code": "sync.parsing.packages",
      "state": "completed",
      "total": 11335,
      "done": 11335,
      "suffix": null
    }
  ],
  "created_resources": [
    "/pulp/api/v3/repositories/rpm/rpm/0197584b-30b7-782f-a5d6-bac21b860e1d/versions/1/"
  ],
  "reserved_resources_record": [
    "prn:rpm.rpmrepository:0197584b-30b7-782f-a5d6-bac21b860e1d",
    "shared:prn:rpm.rpmremote:0197584b-2cc5-7cc4-9813-366ede26314a",
    "shared:prn:core.domain:6a62c031-6b33-414b-a60e-ddd3675aa73d"
  ]
}

du
0       /global/podmandata/pulp-mirror/pulp_storage/tmp/10063@pulp_mirror
0       /global/podmandata/pulp-mirror/pulp_storage/tmp/10111@pulp_mirror
0       /global/podmandata/pulp-mirror/pulp_storage/media/

==========================================================================

pulp task show  --uuid 0197584b-48f7-7ae8-9e80-5d4f3e27ab4a ; du -sh /global/podmandata/pulp-mirror/pulp_storage/tmp/* /global/podmandata/pulp-mirror/pulp_storage/media/
{
  "pulp_href": "/pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/",
  "prn": "prn:core.task:0197584b-48f7-7ae8-9e80-5d4f3e27ab4a",
  "pulp_created": "2025-06-10T05:23:38.360391Z",
  "pulp_last_updated": "2025-06-10T05:23:38.360404Z",
  "state": "completed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "d64bd8ceab924eca9d08092552ffe885",
  "created_by": "/pulp/api/v3/users/1/",
  "unblocked_at": "2025-06-10T05:23:38.375759Z",
  "started_at": "2025-06-10T05:23:38.412487Z",
  "finished_at": "2025-06-10T05:31:27.685569Z",
  "error": null,
  "worker": "/pulp/api/v3/workers/01975849-4467-7fbd-8bdd-1d60a18625a6/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "completed",
      "total": null,
      "done": 5,
      "suffix": null
    },
    {
      "message": "Skipping Packages",
      "code": "sync.skipped.packages",
      "state": "completed",
      "total": 0,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Parsed Advisories",
      "code": "sync.parsing.advisories",
      "state": "completed",
      "total": 3880,
      "done": 3880,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "completed",
      "total": null,
      "done": 11340,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "completed",
      "total": null,
      "done": 15220,
      "suffix": null
    },
    {
      "message": "Parsed Packages",
      "code": "sync.parsing.packages",
      "state": "completed",
      "total": 11335,
      "done": 11335,
      "suffix": null
    }
  ],
  "created_resources": [
    "/pulp/api/v3/repositories/rpm/rpm/0197584b-30b7-782f-a5d6-bac21b860e1d/versions/1/"
  ],
  "reserved_resources_record": [
    "prn:rpm.rpmrepository:0197584b-30b7-782f-a5d6-bac21b860e1d",
    "shared:prn:rpm.rpmremote:0197584b-2cc5-7cc4-9813-366ede26314a",
    "shared:prn:core.domain:6a62c031-6b33-414b-a60e-ddd3675aa73d"
  ]
}

du
0       /global/podmandata/pulp-mirror/pulp_storage/tmp/10063@pulp_mirror
0       /global/podmandata/pulp-mirror/pulp_storage/tmp/10111@pulp_mirror
0       /global/podmandata/pulp-mirror/pulp_storage/media/

Container log

('pulp [d64bd8ceab924eca9d08092552ffe885]: ::ffff:127.0.0.1 - admin [10/Jun/2025:05:31:24 +0000] "GET /pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/ HTTP/1.0" 200 1651 "-" "Pulp-CLI/0.33.0"',)
('pulp [d64bd8ceab924eca9d08092552ffe885]: ::ffff:127.0.0.1 - admin [10/Jun/2025:05:31:26 +0000] "GET /pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/ HTTP/1.0" 200 1651 "-" "Pulp-CLI/0.33.0"',)
('pulp [d64bd8ceab924eca9d08092552ffe885]: ::ffff:127.0.0.1 - admin [10/Jun/2025:05:31:27 +0000] "GET /pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/ HTTP/1.0" 200 1653 "-" "Pulp-CLI/0.33.0"',)
pulp [d64bd8ceab924eca9d08092552ffe885]: pulpcore.tasking.tasks:INFO: Task completed 0197584b-48f7-7ae8-9e80-5d4f3e27ab4a in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 469273082 ÎŒs
('pulp [d64bd8ceab924eca9d08092552ffe885]: ::ffff:127.0.0.1 - admin [10/Jun/2025:05:31:28 +0000] "GET /pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/ HTTP/1.0" 200 1749 "-" "Pulp-CLI/0.33.0"',)
('pulp [9f2e5d73289f4d7d9d24d84b3e1d5b8e]: ::ffff:127.0.0.1 - admin [10/Jun/2025:06:12:01 +0000] "GET /pulp/api/v3/tasks/0197584b-48f7-7ae8-9e80-5d4f3e27ab4a/ HTTP/1.0" 200 1749 "-" "Pulp-CLI/0.33.0"',)
#9

Seen at the first time stating a pulp container.

  Applying container.0029_remove_blob_media_type... OK
  Applying container.0030_enforce_tagged_manifest_reference... OK
  Applying container.0031_replace_charf_with_textf... OK
  Applying container.0032_upload_artifact... OK
  Applying container.0033_raise_warning_for_repair... OK
  Applying container.0034_translate_signed_schema... OK
  Applying container.0035_alter_blob_content_ptr_and_more... OK
  Applying container.0036_containerpushrepository_pending_blobs_manifests... OK
  Applying container.0037_create_pull_through_cache_models... OK
/usr/local/lib/python3.9/site-packages/pulp_container/app/migrations/0038_add_manifest_metadata_fields.py:8: UserWarning: Run 'pulpcore-manager container-handle-image-data' to initialize and expose metadata (i.e., annotations and labels) for all manifests.
  warnings.warn(
  Applying container.0038_add_manifest_metadata_fields... OK
/usr/local/lib/python3.9/site-packages/pulp_container/app/migrations/0039_manifest_data.py:8: UserWarning: Run 'pulpcore-manager container-handle-image-data' to move the manifests' data from artifacts to the new 'data' database field.
  warnings.warn(
  Applying container.0039_manifest_data... OK
  Applying container.0040_add_remote_repo_filter... OK
  Applying container.0041_add_pull_through_pull_permissions... OK
  Applying container.0042_add_manifest_nature_field... OK
/usr/local/lib/python3.9/site-packages/pulp_container/app/migrations/0043_add_os_arch_image_size_manifest_fields.py:7: UserWarning: Run 'pulpcore-manager container-handle-image-data' to update the manifests' os, architecture, and compressed_image_size fields.
  warnings.warn(
  Applying container.0043_add_os_arch_image_size_manifest_fields... OK
  Applying container.0044_add_domain... OK
  Applying container.0045_alter_manifest_compressed_image_size... OK
  Applying core.0126_remoteartifact_failed_at... OK
  Applying core.0127_remove_upstreampulp_pulp_label_select... OK
  Applying core.0128_domain_pulp_labels... OK
  Applying core.0129_content_pulp_labels... OK
  Applying core.0130_upstreampulp_policy... OK
  Applying core.0132_alter_content_options... OK
  Applying core.0131_distribution_checkpoint_publication_checkpoint... OK

    {
      "name": "10138@pulp_mirror",
      "last_heartbeat": "2025-06-10T08:44:52.542665Z",
      "versions": {
        "deb": "3.5.2",
        "gem": "0.7.1",
        "npm": "0.3.3",
        "rpm": "3.30.1",
        "core": "3.80.0",
        "file": "3.80.0",
        "maven": "0.10.1",
        "ostree": "2.4.8",
        "python": "3.15.0",
        "ansible": "0.26.0",
        "certguard": "3.80.0",
        "container": "2.25.1"
      }
#10

Adding another sample:

Creating a repository with --sync-policy mirror_complete
All the steps here go though without an error

  • remote
  • repository
  • publication

But: tmp and artifact directories empty.
So the publication URL points to a rpm that does not exist.

The publication step produces a
"base_url": "http://pulp237.abcgroup.net:30445/pulp/content/MR/SLES12/SP4/2025Q02.00/"

https://pulp237.abcgroup.net:30445/pulp/content/MR/SLES12/SP4/2025Q02.00/repodata/

Index of /pulp/content/MR/SLES12/SP4/2025Q02.00/repodata/
../
29b8fd1bca28fe36e4b23c40187d20fb50b07ae68a3043924765bec5c9b9f6eb-app-icons.tar.gz                   10-Jun-2025 08:55  2.4 kB
29b8fd1bca28fe36e4b23c40187d20fb50b07ae68a3043924765bec5c9b9f6eb-appdata-icons.tar.gz               10-Jun-2025 08:55  2.4 kB
4782090c3a520036cee6324f3c5b5d550fec344ca9f3f802b91e146651ef646e-primary.xml.gz                     10-Jun-2025 08:55  5.0 MB
934fb1afc4bb474e9d7c33a0477e87a6bfe85b91505d7ac8340817765ff0e137-license.tar.gz                     10-Jun-2025 08:55  88.0 kB
980182e3abc99748e15dfe89499b2c7b0df3416c214e94a084599916740db55e-susedata.xml.gz                    10-Jun-2025 08:55  291.5 kB
b098a6338576a991eeab9405d58cf281422e75dc136bb0c3995c4215e42a8305-appdata.xml.gz                     10-Jun-2025 08:55  1.6 kB
b2dcee8e9d50b527aebb014fe5f3f7fb70b0a1769587419b0d1a441bb8b7b9af-filelists.xml.gz                   10-Jun-2025 08:55  9.6 MB
bf73e32441bf420c92cf875338820cc63cd4b9a0643f3fcb8ba8ca3c497bf321-other.xml.gz                       10-Jun-2025 08:55  5.4 MB
e56b47cbe35d05ed4995b1c6b7a6aba0253e65362b38f71b30c04670ddf5c120-updateinfo.xml.gz                  10-Jun-2025 08:55  3.7 MB
repomd.xml                                                                                          10-Jun-2025 08:55  4.3 kB

Index of /pulp/content/MR/SLES12/SP4/2025Q02.00/Packages/
../
a/                                                                                                  10-Jun-2025 08:50  697.0 kB
b/                                                                                                  10-Jun-2025 08:50  115.2 kB
c/                                                                                                  10-Jun-2025 08:50  34.2 kB
d/                                                                                                  10-Jun-2025 08:50  1.2 MB
e/                                                                                                  10-Jun-2025 08:50  130.9 kB
f/                                                                                                  10-Jun-2025 08:50  357.5 kB
g/                                                                                                  10-Jun-2025 08:51  192.5 kB
h/                                                                                                  10-Jun-2025 08:51  43.1 kB
i/                                                                                                  10-Jun-2025 08:51  56.6 kB
j/                                                                                                  10-Jun-2025 08:51  917.1 kB
k/                                                                                                  10-Jun-2025 08:51  3.6 MB
l/                                                                                                  10-Jun-2025 08:53  397.4 kB
m/                                                                                                  10-Jun-2025 08:53  4.7 MB
n/                                                                                                  10-Jun-2025 08:53  2.6 MB
o/                                                                                                  10-Jun-2025 08:53  476.3 kB
p/                                                                                                  10-Jun-2025 08:54  431.3 kB
q/                                                                                                  10-Jun-2025 08:54  175.4 kB
r/                                                                                                  10-Jun-2025 08:54  23.1 kB
s/                                                                                                  10-Jun-2025 08:54  18.6 MB
t/                                                                                                  10-Jun-2025 08:54  63.6 kB
u/                                                                                                  10-Jun-2025 08:54  6.5 MB
v/                                                                                                  10-Jun-2025 08:54  95.5 kB
w/                                                                                                  10-Jun-2025 08:54  812.7 kB
x/                                                                                                  10-Jun-2025 08:55  686.4 kB
y/                                                                                                  10-Jun-2025 08:55  94.5 kB
z/                                                                                                  10-Jun-2025 08:55  1.3 MB

Index of /pulp/content/MR/SLES12/SP4/2025Q02.00/Packages/c/
../
ClusterTools2-3.1.0-19.6.1.noarch.rpm                                                               10-Jun-2025 08:50  119.7 kB
ClusterTools2-3.1.1-19.9.1.noarch.rpm                                                               10-Jun-2025 08:50  135.9 kB
ClusterTools2-3.1.2-19.12.1.noarch.rpm                                                              10-Jun-2025 08:50  143.9 kB
ca-certificates-1_201403302107-6.2.noarch.rpm                                                       10-Jun-2025 08:50  21.5 kB
ca-certificates-mozilla-2.44-12.34.1.noarch.rpm                                                     10-Jun-2025 08:50  183.6 kB
ca-certificates-mozilla-2.56-12.37.1.noarch.rpm                                                     10-Jun-2025 08:50  197.0 kB

Download fails - directory artifacts empty.
DB pointer is fine.

https://pulp237.abcgroup.net:30445/pulp/content/MR/SLES12/SP4/2025Q02.00/Packages/c/ClusterTools2-3.1.0-19.6.1.noarch.rpm

500 Internal Server Error
Server got itself in trouble

Container log on this request:

::ffff:127.0.0.1 [10/Jun/2025:09:17:05 +0000] "GET /pulp/content/MR/SLES12/SP4/2025Q02.00/Packages/c/ HTTP/1.0" 200 52985 "https://pulp237.abcgroup.net:30445/pulp/content/MR/SLES12/SP4/2025Q02.00/Packages/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"
[2025-06-10 09:18:02 +0000] [10142] [ERROR] Error handling request from ::ffff:127.0.0.1
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 510, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_app.py", line 569, in _handle
    return await handler(request)
  File "/usr/local/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 27, in guid
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 58, in authenticate
    return await handler(request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 290, in stream_content
    return await self._match_and_stream(path, request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 806, in _match_and_stream
    return await self._serve_content_artifact(ca, headers, request)
  File "/usr/local/lib/python3.9/site-packages/pulpcore/content/handler.py", line 1163, in _serve_content_artifact
    content_length = artifact_file.size
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/files.py", line 74, in size
    return self.storage.size(self.name)
  File "/usr/local/lib/python3.9/site-packages/django/core/files/storage/filesystem.py", line 182, in size
    return os.path.getsize(self.path(name))
  File "/usr/lib64/python3.9/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pulp/tmp/artifact/db/c501621ec581457a930b1a5911a68549b821e7983bb984315dc6e27a81c0e8'
::ffff:127.0.0.1 [10/Jun/2025:09:18:02 +0000] "GET /pulp/content/MR/SLES12/SP4/2025Q02.00/Packages/c/ClusterTools2-3.1.0-19.6.1.noarch.rpm HTTP/1.0" 500 317 "https://pulp237.abcgroup.net:30445/pulp/content/MR/SLES12/SP4/2025Q02.00/Packages/c/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"


{
  "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/01975908-4ec2-74f8-92f5-ee55c08cd081/",
  "prn": "prn:rpm.rpmremote:01975908-4ec2-74f8-92f5-ee55c08cd081",
  "pulp_created": "2025-06-10T08:50:06.147362Z",
  "pulp_last_updated": "2025-06-10T08:50:06.147381Z",
  "name": "remote-active-P-2025Q02.00-ABC-SLES12-SP4",
  "url": "https://ltnomadcli01.abcgroup.net:30082/pulp/content/MR/SLES12/SP4/2025Q02.00/active-P-2025Q02.00-ABC-SLES12-SP4/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": false,
  "proxy_url": null,
  "pulp_labels": {},
  "download_concurrency": 6,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": 7200.0,
  "connect_timeout": null,
  "sock_connect_timeout": 720.0,
  "sock_read_timeout": 720.0,
  "headers": null,
  "rate_limit": null,
  "hidden_fields": [
    {
      "name": "client_key",
      "is_set": false
    },
    {
      "name": "proxy_username",
      "is_set": false
    },
    {
      "name": "proxy_password",
      "is_set": false
    },
    {
      "name": "username",
      "is_set": false
    },
    {
      "name": "password",
      "is_set": false
    }
  ],
  "sles_auth_token": null
}
{
  "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/01975908-5197-78f6-99f3-9a53c743ae5c/",
  "prn": "prn:rpm.rpmrepository:01975908-5197-78f6-99f3-9a53c743ae5c",
  "pulp_created": "2025-06-10T08:50:06.872308Z",
  "pulp_last_updated": "2025-06-10T08:50:06.878873Z",
  "versions_href": "/pulp/api/v3/repositories/rpm/rpm/01975908-5197-78f6-99f3-9a53c743ae5c/versions/",
  "pulp_labels": {},
  "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/01975908-5197-78f6-99f3-9a53c743ae5c/versions/0/",
  "name": "active-P-2025Q02.00-ABC-SLES12-SP4",
  "description": "active-P-2025Q02.00-ABC-SLES12-SP4",
  "retain_repo_versions": null,
  "remote": "/pulp/api/v3/remotes/rpm/rpm/01975908-4ec2-74f8-92f5-ee55c08cd081/",
  "autopublish": false,
  "metadata_signing_service": null,
  "package_signing_service": null,
  "package_signing_fingerprint": "",
  "retain_package_versions": 0,
  "checksum_type": null,
  "metadata_checksum_type": null,
  "package_checksum_type": null,
  "sqlite_metadata": false,
  "repo_config": {},
  "compression_type": null,
  "layout": null
}
REPO_HREF = /pulp/api/v3/repositories/rpm/rpm/01975908-5197-78f6-99f3-9a53c743ae5c/
repo
======================================
Started background task /pulp/api/v3/tasks/01975908-6902-7342-b838-0c691a3a9521/
..................................................................................................................................................................................................................................................................................Done.
0       /global/podmandata/pulp-mirror/pulp_storage/media
8.0K    /global/podmandata/pulp-mirror/pulp_storage/scripts
0       /global/podmandata/pulp-mirror/pulp_storage/tmp
sync
=====================================



pulp task show  --uuid 01975908-6902-7342-b838-0c691a3a9521 ; du -sh /global/podmandata/pulp-mirror/pulp_storage/tmp/* /global/podmandata/pulp-mirror/pulp_storage/media/
{
  "pulp_href": "/pulp/api/v3/tasks/01975908-6902-7342-b838-0c691a3a9521/",
  "prn": "prn:core.task:01975908-6902-7342-b838-0c691a3a9521",
  "pulp_created": "2025-06-10T08:50:12.866931Z",
  "pulp_last_updated": "2025-06-10T08:50:12.866943Z",
  "state": "completed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "04b5e0c8090d4bf4a1c06b2825805905",
  "created_by": "/pulp/api/v3/users/1/",
  "unblocked_at": "2025-06-10T08:50:12.879953Z",
  "started_at": "2025-06-10T08:50:12.919477Z",
  "finished_at": "2025-06-10T08:55:27.029835Z",
  "error": null,
  "worker": "/pulp/api/v3/workers/019758fe-d0c0-7951-b0dd-de406c3fbfdd/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "completed",
      "total": null,
      "done": 10,
      "suffix": null
    },
    {
      "message": "Skipping Packages",
      "code": "sync.skipped.packages",
      "state": "completed",
      "total": 0,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Parsed Advisories",
      "code": "sync.parsing.advisories",
      "state": "completed",
      "total": 3880,
      "done": 3880,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "completed",
      "total": null,
      "done": 11340,
      "suffix": null
    },
    {
      "message": "Un-Associating Content",
      "code": "unassociating.content",
      "state": "completed",
      "total": null,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "completed",
      "total": null,
      "done": 15220,
      "suffix": null
    },
    {
      "message": "Parsed Packages",
      "code": "sync.parsing.packages",
      "state": "completed",
      "total": 11335,
      "done": 11335,
      "suffix": null
    }
  ],
  "created_resources": [
    "/pulp/api/v3/repositories/rpm/rpm/01975908-5197-78f6-99f3-9a53c743ae5c/versions/1/",
    "/pulp/api/v3/publications/rpm/rpm/0197590d-c4ec-7e87-a792-15fe1d08dc6a/"
  ],
  "reserved_resources_record": [
    "prn:rpm.rpmrepository:01975908-5197-78f6-99f3-9a53c743ae5c",
    "shared:prn:rpm.rpmremote:01975908-4ec2-74f8-92f5-ee55c08cd081",
    "shared:prn:core.domain:d05be0b3-6b86-4271-9881-737a9fcd3209"
  ]
}

0       /global/podmandata/pulp-mirror/pulp_storage/tmp/10063@pulp_mirror
0       /global/podmandata/pulp-mirror/pulp_storage/tmp/10110@pulp_mirror
0       /global/podmandata/pulp-mirror/pulp_storage/media/

Container log at end of repo build.

('pulp [04b5e0c8090d4bf4a1c06b2825805905]: ::ffff:127.0.0.1 - admin [10/Jun/2025:08:55:24 +0000] "GET /pulp/api/v3/tasks/01975908-6902-7342-b838-0c691a3a9521/ HTTP/1.0" 200 1778 "-" "Pulp-CLI/0.33.0"',)
('pulp [04b5e0c8090d4bf4a1c06b2825805905]: ::ffff:127.0.0.1 - admin [10/Jun/2025:08:55:25 +0000] "GET /pulp/api/v3/tasks/01975908-6902-7342-b838-0c691a3a9521/ HTTP/1.0" 200 1921 "-" "Pulp-CLI/0.33.0"',)
('pulp [04b5e0c8090d4bf4a1c06b2825805905]: ::ffff:127.0.0.1 - admin [10/Jun/2025:08:55:26 +0000] "GET /pulp/api/v3/tasks/01975908-6902-7342-b838-0c691a3a9521/ HTTP/1.0" 200 1921 "-" "Pulp-CLI/0.33.0"',)
pulp [04b5e0c8090d4bf4a1c06b2825805905]: pulpcore.tasking.tasks:INFO: Task completed 01975908-6902-7342-b838-0c691a3a9521 in domain: default, task_type: pulp_rpm.app.tasks.synchronizing.synchronize, immediate: False, deferred: True, execution_time: 314110358 ÎŒs
('pulp [04b5e0c8090d4bf4a1c06b2825805905]: ::ffff:127.0.0.1 - admin [10/Jun/2025:08:55:28 +0000] "GET /pulp/api/v3/tasks/01975908-6902-7342-b838-0c691a3a9521/ HTTP/1.0" 200 1948 "-" "Pulp-CLI/0.33.0"',)
('pulp [2576d5d793144974a7582410372a3ce9]: ::ffff:127.0.0.1 - admin [10/Jun/2025:08:55:36 +0000] "GET /pulp/api/v3/tasks/01975908-6902-7342-b838-0c691a3a9521/ HTTP/1.0" 200 1948 "-" "Pulp-CLI/0.33.0"',)
#11

This is pretty bizarre. Artifacts are never stored in /var/lib/pulp/tmp/artifacts/, they’re stored in /var/lib/pulp/media/artifacts/ (at least they’re intended to be?)

I don’t suppose you’ve tweaked the values of MEDIA_ROOT or DEPLOY_ROOT?

#12

Hi dralley

Yes, they are untouched.

#13

Can you share a dump of all the settings? I’ll try to reproduce.
From you container:

dynaconf list --json
2 Likes
#14

Thank you pedro-psb.

The compressed json, hostname and password obscured.

{“ABSOLUTE_URL_OVERRIDES”:{},“ACCESS_POLICIES”:{},“ADMINS”:[],“ALLOWED_CONTENT_CHECKSUMS”:[“sha224”,“sha256”,“sha384”,“sha512”],“ALLOWED_EXPORT_PATHS”:[],“ALLOWED_HOSTS”:["*"],“ALLOWED_IMPORT_PATHS”:[],“ALLOW_AUTOMATIC_UNSAFE_ADVISORY_CONFLICT_RESOLUTION”:false,“ANALYTICS”:true,“ANSIBLE_API_HOSTNAME”:“http://mypulp.net.net:30445”,“ANSIBLE_AUTHENTICATION_CLASSES”:[“pulpcore.app.authentication.BasicAuthentication”,“rest_framework.authentication.SessionAuthentication”],“ANSIBLE_COLLECT_DOWNLOAD_COUNT”:false,“ANSIBLE_COLLECT_DOWNLOAD_LOG”:false,“ANSIBLE_CONTENT_HOSTNAME”:“http://mypulp.net.net:30445/pulp/content",“ANSIBLE_DEFAULT_DISTRIBUTION_PATH”:null,“ANSIBLE_PERMISSION_CLASSES”:[“pulpcore.app.access_policy.AccessPolicyFromDB”],“ANSIBLE_SIGNATURE_REQUIRE_VERIFICATION”:true,“ANSIBLE_SIGNING_TASK_LIMITER”:10,“ANSIBLE_URL_NAMESPACE”:"",“API_APP_TTL”:120,“API_ROOT”:"/pulp/",“API_ROOT_REWRITE_HEADER”:null,“API_WORKERS”:2,“API_WORKERS_MAX_REQUESTS”:0,“API_WORKERS_MAX_REQUESTS_JITTER”:0,“APPEND_SLASH”:true,“APT_BY_HASH”:false,“AUTHENTICATION_BACKENDS”:[“django.contrib.auth.backends.ModelBackend”,“pulpcore.backends.ObjectRolePermissionBackend”],“AUTHENTICATION_JSON_HEADER”:"",“AUTHENTICATION_JSON_HEADER_JQ_FILTER”:"",“AUTHENTICATION_JSON_HEADER_OPENAPI_SECURITY_SCHEME”:{},“AUTH_PASSWORD_VALIDATORS”:[{“NAME”:“django.contrib.auth.password_validation.UserAttributeSimilarityValidator”},{“NAME”:“django.contrib.auth.password_validation.MinimumLengthValidator”},{“NAME”:“django.contrib.auth.password_validation.CommonPasswordValidator”},{“NAME”:“django.contrib.auth.password_validation.NumericPasswordValidator”}],“AUTH_USER_MODEL”:“auth.User”,“BASE_DIR”:"/usr/local/lib/python3.9/site-packages/pulpcore/app",“CACHES”:{“default”:{“BACKEND”:“django.core.cache.backends.locmem.LocMemCache”}},“CACHE_ENABLED”:false,“CACHE_MIDDLEWARE_ALIAS”:“default”,“CACHE_MIDDLEWARE_KEY_PREFIX”:"",“CACHE_MIDDLEWARE_SECONDS”:600,“CACHE_SETTINGS”:{“EXPIRES_TTL”:600},“CHUNKED_UPLOAD_DIR”:“upload”,“COLLECTION_SIGNING_SCRIPT”:"/var/lib/pulp/scripts/vault_collection_sign_mirror.sh",“COLLECTION_SIGNING_SERVICE”:“abc_collection_signer_vault”,“CONTENT_APP_TTL”:30,“CONTENT_ORIGIN”:“http://mypulp.net.net:30445”,“CONTENT_PATH_PREFIX”:"/pulp/content/",“CONTENT_WORKERS”:2,“CONTENT_WORKERS_MAX_REQUESTS”:0,“CONTENT_WORKERS_MAX_REQUESTS_JITTER”:0,“CSRF_COOKIE_AGE”:31449600,“CSRF_COOKIE_DOMAIN”:null,“CSRF_COOKIE_HTTPONLY”:false,“CSRF_COOKIE_MASKED”:false,“CSRF_COOKIE_NAME”:“csrftoken”,“CSRF_COOKIE_PATH”:"/",“CSRF_COOKIE_SAMESITE”:“Lax”,“CSRF_COOKIE_SECURE”:false,“CSRF_FAILURE_VIEW”:“django.views.csrf.csrf_failure”,“CSRF_HEADER_NAME”:“HTTP_X_CSRFTOKEN”,“CSRF_TRUSTED_ORIGINS”:[],“CSRF_USE_SESSIONS”:false,“DATABASES”:{“default”:{“ATOMIC_REQUESTS”:false,“AUTOCOMMIT”:true,“CONN_HEALTH_CHECKS”:false,“CONN_MAX_AGE”:0,“ENGINE”:“django.db.backends.postgresql”,“HOST”:"",“NAME”:“pulp”,“OPTIONS”:{},“PASSWORD”:"",“PORT”:"",“TEST”:{“CHARSET”:null,“COLLATION”:null,“MIGRATE”:true,“MIRROR”:null,“NAME”:null},“TIME_ZONE”:null,“USER”:“pulp”}},“DATABASE_ROUTERS”:[],“DATA_UPLOAD_MAX_MEMORY_SIZE”:2621440,“DATA_UPLOAD_MAX_NUMBER_FIELDS”:1000,“DATA_UPLOAD_MAX_NUMBER_FILES”:100,“DATETIME_FORMAT”:"N j, Y, P”,“DATETIME_INPUT_FORMATS”:["%Y-%m-%d %H:%M:%S","%Y-%m-%d %H:%M:%S.%f","%Y-%m-%d %H:%M","%m/%d/%Y %H:%M:%S","%m/%d/%Y %H:%M:%S.%f","%m/%d/%Y %H:%M","%m/%d/%y %H:%M:%S","%m/%d/%y %H:%M:%S.%f","%m/%d/%y %H:%M"],“DATE_FORMAT”:“N j, Y”,“DATE_INPUT_FORMATS”:["%Y-%m-%d","%m/%d/%Y","%m/%d/%y","%b %d %Y","%b %d, %Y","%d %b %Y","%d %b, %Y","%B %d %Y","%B %d, %Y","%d %B %Y","%d %B, %Y"],“DB_ENCRYPTION_KEY”:"/etc/pulp/certs/database_fields.symmetric.key",“DEBUG”:false,“DEBUG_PROPAGATE_EXCEPTIONS”:false,“DECIMAL_SEPARATOR”:".",“DEFAULT_ADMIN_PASSWORD”:“pulp-pw”,“DEFAULT_AUTO_FIELD”:“django.db.models.AutoField”,“DEFAULT_CHARSET”:“utf-8”,“DEFAULT_EXCEPTION_REPORTER”:“django.views.debug.ExceptionReporter”,“DEFAULT_EXCEPTION_REPORTER_FILTER”:“django.views.debug.SafeExceptionReporterFilter”,“DEFAULT_FILE_STORAGE”:“pulpcore.app.models.storage.FileSystem”,“DEFAULT_FROM_EMAIL”:“webmaster@localhost”,“DEFAULT_INDEX_TABLESPACE”:"",“DEFAULT_SETTINGS_PATHS”:[“settings.py”,“settings.toml”,“settings.tml”,“settings.yaml”,“settings.yml”,“settings.ini”,“settings.conf”,“settings.properties”,“settings.json”,".secrets.py",".secrets.toml",".secrets.tml",".secrets.yaml",".secrets.yml",".secrets.ini",".secrets.conf",".secrets.properties",".secrets.json"],“DEFAULT_TABLESPACE”:"",“DEFAULT_ULN_SERVER_BASE_URL”:“https://linux-update.oracle.com/",“DEPLOY_ROOT”:"/var/lib/pulp",“DISALLOWED_USER_AGENTS”:[],“DOMAIN_ENABLED”:false,“DRF_ACCESS_POLICY”:{“reusable_conditions”:[“pulpcore.app.global_access_conditions”,“pulp_ansible.app.global_access_conditions”,“pulp_container.app.global_access_conditions”,“pulp_python.app.global_access_conditions”,“pulp_rpm.app.access_policy”]},“DYNACONF”:"<dynaconf.base.LazySettings object at 0x7f8da90160a0>”,“EMAIL_BACKEND”:“django.core.mail.backends.smtp.EmailBackend”,“EMAIL_HOST”:“localhost”,“EMAIL_HOST_PASSWORD”:"",“EMAIL_HOST_USER”:"",“EMAIL_PORT”:25,“EMAIL_SSL_CERTFILE”:null,“EMAIL_SSL_KEYFILE”:null,“EMAIL_SUBJECT_PREFIX”:"[Django] “,“EMAIL_TIMEOUT”:null,“EMAIL_USE_LOCALTIME”:false,“EMAIL_USE_SSL”:false,“EMAIL_USE_TLS”:false,“FILE_UPLOAD_DIRECTORY_PERMISSIONS”:null,“FILE_UPLOAD_HANDLERS”:[“pulpcore.app.files.HashingFileUploadHandler”],“FILE_UPLOAD_MAX_MEMORY_SIZE”:2621440,“FILE_UPLOAD_PERMISSIONS”:420,“FILE_UPLOAD_TEMP_DIR”:”/var/lib/pulp/tmp",“FIRST_DAY_OF_WEEK”:0,“FIXTURE_DIRS”:[],“FLATPAK_INDEX”:false,“FORBIDDEN_CHECKSUM_WARNINGS”:true,“FORCE_IGNORE_MISSING_PACKAGE_INDICES”:false,“FORCE_SCRIPT_NAME”:null,“FORMAT_MODULE_PATH”:null,“FORM_RENDERER”:“django.forms.renderers.DjangoTemplates”,“GUNICORN_RELOAD”:false,“GUNICORN_TIMEOUT”:90,“HIDE_GUARDED_DISTRIBUTIONS”:false,“HTTPS”:true,“IGNORABLE_404_URLS”:[],“IMPORT_WORKERS_PERCENT”:100,“INSTALLED_APPS”:[“django.contrib.auth”,“django.contrib.contenttypes”,“django.contrib.sessions”,“django.contrib.messages”,“django.contrib.staticfiles”,“django.contrib.postgres”,“import_export”,“django_filters”,“django_guid”,“drf_spectacular”,“rest_framework”,“pulpcore.app”,“storages”,“pulp_ansible.app.PulpAnsiblePluginAppConfig”,“pulp_container.app.PulpContainerPluginAppConfig”,“pulp_deb.app.PulpDebPluginAppConfig”,“pulp_gem.app.PulpGemPluginAppConfig”,“pulp_maven.app.PulpMavenPluginAppConfig”,“pulp_npm.app.PulpNpmPluginAppConfig”,“pulp_ostree.app.PulpOstreePluginAppConfig”,“pulp_python.app.PulpPythonPluginAppConfig”,“pulp_rpm.app.PulpRpmPluginAppConfig”,“pulp_certguard.app.PulpCertGuardPluginAppConfig”,“pulp_file.app.PulpFilePluginAppConfig”,“django_readonly_field”],“INTERNAL_IPS”:[],“KAFKA_BOOTSTRAP_SERVERS”:null,“KAFKA_PRODUCER_POLL_TIMEOUT”:0.1,“KAFKA_SASL_MECHANISM”:null,“KAFKA_SASL_PASSWORD”:null,“KAFKA_SASL_USERNAME”:null,“KAFKA_SECURITY_PROTOCOL”:“plaintext”,“KAFKA_SSL_CA_PEM”:null,“KAFKA_TASKS_STATUS_PRODUCER_SYNC_ENABLED”:false,“KAFKA_TASKS_STATUS_TOPIC”:“pulpcore.tasking.status”,“KEEP_CHANGELOG_LIMIT”:10,“LANGUAGES”:[[“af”,“Afrikaans”],[“ar”,“Arabic”],[“ar-dz”,“Algerian Arabic”],[“ast”,“Asturian”],[“az”,“Azerbaijani”],[“bg”,“Bulgarian”],[“be”,“Belarusian”],[“bn”,“Bengali”],[“br”,“Breton”],[“bs”,“Bosnian”],[“ca”,“Catalan”],[“ckb”,“Central Kurdish (Sorani)”],[“cs”,“Czech”],[“cy”,“Welsh”],[“da”,“Danish”],[“de”,“German”],[“dsb”,“Lower Sorbian”],[“el”,“Greek”],[“en”,“English”],[“en-au”,“Australian English”],[“en-gb”,“British English”],[“eo”,“Esperanto”],[“es”,“Spanish”],[“es-ar”,“Argentinian Spanish”],[“es-co”,“Colombian Spanish”],[“es-mx”,“Mexican Spanish”],[“es-ni”,“Nicaraguan Spanish”],[“es-ve”,“Venezuelan Spanish”],[“et”,“Estonian”],[“eu”,“Basque”],[“fa”,“Persian”],[“fi”,“Finnish”],[“fr”,“French”],[“fy”,“Frisian”],[“ga”,“Irish”],[“gd”,“Scottish Gaelic”],[“gl”,“Galician”],[“he”,“Hebrew”],[“hi”,“Hindi”],[“hr”,“Croatian”],[“hsb”,“Upper Sorbian”],[“hu”,“Hungarian”],[“hy”,“Armenian”],[“ia”,“Interlingua”],[“id”,“Indonesian”],[“ig”,“Igbo”],[“io”,“Ido”],[“is”,“Icelandic”],[“it”,“Italian”],[“ja”,“Japanese”],[“ka”,“Georgian”],[“kab”,“Kabyle”],[“kk”,“Kazakh”],[“km”,“Khmer”],[“kn”,“Kannada”],[“ko”,“Korean”],[“ky”,“Kyrgyz”],[“lb”,“Luxembourgish”],[“lt”,“Lithuanian”],[“lv”,“Latvian”],[“mk”,“Macedonian”],[“ml”,“Malayalam”],[“mn”,“Mongolian”],[“mr”,“Marathi”],[“ms”,“Malay”],[“my”,“Burmese”],[“nb”,“Norwegian BokmĂ„l”],[“ne”,“Nepali”],[“nl”,“Dutch”],[“nn”,“Norwegian Nynorsk”],[“os”,“Ossetic”],[“pa”,“Punjabi”],[“pl”,“Polish”],[“pt”,“Portuguese”],[“pt-br”,“Brazilian Portuguese”],[“ro”,“Romanian”],[“ru”,“Russian”],[“sk”,“Slovak”],[“sl”,“Slovenian”],[“sq”,“Albanian”],[“sr”,“Serbian”],[“sr-latn”,“Serbian Latin”],[“sv”,“Swedish”],[“sw”,“Swahili”],[“ta”,“Tamil”],[“te”,“Telugu”],[“tg”,“Tajik”],[“th”,“Thai”],[“tk”,“Turkmen”],[“tr”,“Turkish”],[“tt”,“Tatar”],[“udm”,“Udmurt”],[“uk”,“Ukrainian”],[“ur”,“Urdu”],[“uz”,“Uzbek”],[“vi”,“Vietnamese”],[“zh-hans”,“Simplified Chinese”],[“zh-hant”,“Traditional Chinese”]],“LANGUAGES_BIDI”:[“he”,“ar”,“ar-dz”,“ckb”,“fa”,“ur”],“LANGUAGE_CODE”:“en-us”,“LANGUAGE_COOKIE_AGE”:null,“LANGUAGE_COOKIE_DOMAIN”:null,“LANGUAGE_COOKIE_HTTPONLY”:false,“LANGUAGE_COOKIE_NAME”:“django_language”,“LANGUAGE_COOKIE_PATH”:"/",“LANGUAGE_COOKIE_SAMESITE”:null,“LANGUAGE_COOKIE_SECURE”:false,“LOAD_DOTENV”:false,“LOCALE_PATHS”:[],“LOGGING”:{“disable_existing_loggers”:false,“filters”:{“correlation_id”:{"()":“django_guid.log_filters.CorrelationId”}},“formatters”:{“simple”:{“format”:“pulp [%(correlation_id)s]: %(name)s:%(levelname)s: %(message)s”}},“handlers”:{“collection_import”:{“class”:“pulp_ansible.app.logutils.CollectionImportHandler”,“formatter”:“simple”,“level”:“DEBUG”},“console”:{“class”:“logging.StreamHandler”,“filters”:[“correlation_id”],“formatter”:“simple”}},“loggers”:{"":{“filters”:[“correlation_id”],“handlers”:[“console”],“level”:“INFO”},“django_guid”:{“handlers”:[“console”],“level”:“WARNING”,“propagate”:false},“pulp_ansible.app.tasks.upload.process_collection_artifact”:{“handlers”:[“collection_import”],“level”:“INFO”,“propagate”:false}},“version”:1},“LOGGING_CONFIG”:“logging.config.dictConfig”,“LOGIN_REDIRECT_URL”:"/accounts/profile/",“LOGIN_URL”:"/accounts/login/",“LOGOUT_REDIRECT_URL”:null,“MANAGERS”:[],“MAX_PARALLEL_SIGNING_TASKS”:10,“MEDIA_ROOT”:"",“MEDIA_URL”:"/",“MESSAGE_STORAGE”:“django.contrib.messages.storage.fallback.FallbackStorage”,“MIDDLEWARE”:[“django_guid.middleware.guid_middleware”,“django.middleware.security.SecurityMiddleware”,“whitenoise.middleware.WhiteNoiseMiddleware”,“django.contrib.sessions.middleware.SessionMiddleware”,“django.middleware.common.CommonMiddleware”,“django.middleware.csrf.CsrfViewMiddleware”,“django.contrib.auth.middleware.AuthenticationMiddleware”,“django.contrib.messages.middleware.MessageMiddleware”,“django.middleware.clickjacking.XFrameOptionsMiddleware”,“pulpcore.middleware.DomainMiddleware”,“pulpcore.middleware.APIRootRewriteMiddleware”],“MIGRATION_MODULES”:{},“MONTH_DAY_FORMAT”:“F j”,“NOCACHE_LIST”:[“repomd.xml”,“repomd.xml.asc”,“repomd.xml.key”],“NUMBER_GROUPING”:0,“OPTIONAL_APPS”:[“crispy_forms”,“django_extensions”,“storages”],“ORIGINAL_SETTINGS_MODULE”:“pulpcore.app.settings”,“ORPHAN_PROTECTION_TIME”:1440,“OTEL_ENABLED”:false,“PASSWORD_HASHERS”:[“django.contrib.auth.hashers.PBKDF2PasswordHasher”,“django.contrib.auth.hashers.PBKDF2SHA1PasswordHasher”,“django.contrib.auth.hashers.Argon2PasswordHasher”,“django.contrib.auth.hashers.BCryptSHA256PasswordHasher”,“django.contrib.auth.hashers.ScryptPasswordHasher”],“PASSWORD_RESET_TIMEOUT”:259200,“POST_HOOKS”:["<function otel_middleware_hook at 0x7f8da8fb29d0>"],“PREPEND_WWW”:false,“PRUNE_WORKERS_MAX”:5,“PULP_HTTPS”:true,“PULP_SIGNING_KEY_FINGERPRINT”:“22718DAFA89C4AA071E950C51E7E5F9F4F33F89A”,“PYPI_API_HOSTNAME”:“https://pulp_mirror”,“PYTHON_GROUP_UPLOADS”:false,“REDIRECT_TO_OBJECT_STORAGE”:true,“REDIS_DB”:0,“REDIS_HOST”:null,“REDIS_PASSWORD”:null,“REDIS_PORT”:null,“REDIS_SSL”:false,“REDIS_SSL_CA_CERTS”:null,“REDIS_URL”:null,“REMOTE_CONTENT_FETCH_FAILURE_COOLDOWN”:300,“REMOTE_USER_ENVIRON_NAME”:“REMOTE_USER”,“REST_FRAMEWORK”:{“DEFAULT_AUTHENTICATION_CLASSES”:[“pulpcore.app.authentication.BasicAuthentication”,“rest_framework.authentication.SessionAuthentication”],“DEFAULT_FILTER_BACKENDS”:[“pulpcore.filters.PulpFilterBackend”],“DEFAULT_PAGINATION_CLASS”:“rest_framework.pagination.LimitOffsetPagination”,“DEFAULT_PERMISSION_CLASSES”:[“pulpcore.app.access_policy.AccessPolicyFromDB”],“DEFAULT_SCHEMA_CLASS”:“pulpcore.openapi.PulpAutoSchema”,“DEFAULT_VERSIONING_CLASS”:“rest_framework.versioning.URLPathVersioning”,“PAGE_SIZE”:100,“UPLOADED_FILES_USE_URL”:false,“URL_FIELD_NAME”:“pulp_href”},“ROOT_URLCONF”:“pulpcore.app.urls”,“RPM_METADATA_USE_REPO_PACKAGE_TIME”:false,“SECRET_KEY”:“SECRET”,“SECRET_KEY_FALLBACKS”:[],“SECURE_CONTENT_TYPE_NOSNIFF”:true,“SECURE_CROSS_ORIGIN_OPENER_POLICY”:“same-origin”,“SECURE_HSTS_INCLUDE_SUBDOMAINS”:false,“SECURE_HSTS_PRELOAD”:false,“SECURE_HSTS_SECONDS”:0,“SECURE_PROXY_SSL_HEADER”:null,“SECURE_REDIRECT_EXEMPT”:[],“SECURE_REFERRER_POLICY”:“same-origin”,“SECURE_SSL_HOST”:null,“SECURE_SSL_REDIRECT”:false,“SERVER_EMAIL”:“root@localhost”,“SESSION_CACHE_ALIAS”:“default”,“SESSION_COOKIE_AGE”:1209600,“SESSION_COOKIE_DOMAIN”:null,“SESSION_COOKIE_HTTPONLY”:true,“SESSION_COOKIE_NAME”:“sessionid”,“SESSION_COOKIE_PATH”:"/",“SESSION_COOKIE_SAMESITE”:“Lax”,“SESSION_COOKIE_SECURE”:false,“SESSION_ENGINE”:“django.contrib.sessions.backends.db”,“SESSION_EXPIRE_AT_BROWSER_CLOSE”:false,“SESSION_FILE_PATH”:null,“SESSION_SAVE_EVERY_REQUEST”:false,“SESSION_SERIALIZER”:“django.contrib.sessions.serializers.JSONSerializer”,“SETTINGS”:"/etc/pulp/settings.py",“SHELL_PLUS_IMPORTS”:[“from pulpcore.app.util import get_domain, get_domain_pk, set_domain, get_url, extract_pk”,“from pulpcore.tasking.tasks import dispatch, cancel_task, wakeup_worker”],“SHORT_DATETIME_FORMAT”:“m/d/Y P”,“SHORT_DATE_FORMAT”:“m/d/Y”,“SIGNING_BACKEND”:“django.core.signing.TimestampSigner”,“SIGNING_KEY_FINGERPRINT”:“22718DAFA89C4AA071E950C51E7E5F9F4F33F89A”,“SILENCED_SYSTEM_CHECKS”:[],“SOCIAL_AUTH_JSONFIELD_CUSTOM”:“django.db.models.JSONField”,“SOCIAL_AUTH_POSTGRES_JSONFIELD”:true,“SOCIAL_AUTH_URL_NAMESPACE”:“social”,“SOLVER_DEBUG_LOGS”:true,“SPECTACULAR_SETTINGS”:{“COMPONENT_NO_READ_ONLY_REQUIRED”:true,“COMPONENT_SPLIT_REQUEST”:true,“CONTACT”:{“email":"pulp-list@redhat.com”,“name”:“Pulp Team”,“url”:“https://pulpproject.org”},“DEFAULT_GENERATOR_CLASS”:“pulpcore.openapi.PulpSchemaGenerator”,“DEFAULT_SCHEMA_CLASS”:“pulpcore.openapi.PulpAutoSchema”,“DESCRIPTION”:“Fetch, Upload, Organize, and Distribute Software Packages”,“DISABLE_ERRORS_AND_WARNINGS”:true,“ENUM_ADD_EXPLICIT_BLANK_NULL_CHOICE”:false,“GENERIC_ADDITIONAL_PROPERTIES”:null,“LICENSE”:{“name”:“GPLv2+”,“url”:“https://raw.githubusercontent.com/pulp/pulpcore/master/LICENSE"},“POSTPROCESSING_HOOKS”:[“drf_spectacular.hooks.postprocess_schema_enums”,“pulpcore.openapi.hooks.add_info_hook”],“SERVE_URLCONF”:“pulpcore.app.urls”,“TITLE”:"Pulp 3 API”,“VERSION”:“v3”},“STATICFILES_DIRS”:[],“STATICFILES_FINDERS”:[“django.contrib.staticfiles.finders.FileSystemFinder”,“django.contrib.staticfiles.finders.AppDirectoriesFinder”],“STATICFILES_STORAGE”:“django.contrib.staticfiles.storage.StaticFilesStorage”,“STATIC_ROOT”:"/var/lib/operator/static/",“STATIC_URL”:"/assets/",“STORAGES”:{“default”:{“BACKEND”:“pulpcore.app.models.storage.FileSystem”},“staticfiles”:{“BACKEND”:“django.contrib.staticfiles.storage.StaticFilesStorage”}},“TASK_DIAGNOSTICS”:[],“TASK_GRACE_INTERVAL”:600,“TASK_PROTECTION_TIME”:0,“TEMPLATES”:[{“APP_DIRS”:true,“BACKEND”:“django.template.backends.django.DjangoTemplates”,“DIRS”:[“PosixPath(’/usr/local/lib/python3.9/site-packages/pulpcore/app/templates’)”],“OPTIONS”:{“context_processors”:[“django.template.context_processors.debug”,“django.template.context_processors.request”,“django.contrib.auth.context_processors.auth”,“django.contrib.messages.context_processors.messages”]}}],“TEST_NON_SERIALIZED_APPS”:[],“TEST_RUNNER”:“django.test.runner.DiscoverRunner”,“THOUSAND_SEPARATOR”:",",“TIME_FORMAT”:“P”,“TIME_INPUT_FORMATS”:["%H:%M:%S","%H:%M:%S.%f","%H:%M"],“TIME_ZONE”:“UTC”,“TMPFILE_PROTECTION_TIME”:0,“TOKEN_AUTH_DISABLED”:true,“UI”:“https://github.com/pulp/pulp-ui/releases/download/v0.1.22/pulp-ui-2025-01-16.tar.gz",“UPLOAD_PROTECTION_TIME”:0,“USE_DEPRECATED_PYTZ”:false,“USE_I18N”:[“USE_I18N”,true],“USE_L10N”:true,“USE_THOUSAND_SEPARATOR”:false,“USE_TZ”:true,“USE_X_FORWARDED_HOST”:false,“USE_X_FORWARDED_PORT”:false,“UVLOOP_ENABLED”:false,“V3_API_ROOT”:"/pulp/api/v3/",“V3_API_ROOT_NO_FRONT_SLASH”:“pulp/api/v3/”,“V3_DOMAIN_API_ROOT”:"/pulp/slug:pulp_domain/api/v3/”,“V3_DOMAIN_API_ROOT_NO_FRONT_SLASH”:“pulp/slug:pulp_domain/api/v3/”,“VAULT_ADDR”:“https://myvault.net:8200”,“VAULT_GPG_HOST”:“https://myvault.net:8200/v1/gpg/keys/pulp3",“VAULT_GPG_SIGNER”:“https://myvault.net:8200/v1/gpg/sign/pulp3/sha2-512”,“VAULT_TOKEN”:“aHZzLkNBRVNJRW5qZDFSUnRVcFVGZDU3cDE3eDAxWmMxREV4ZVR5TGJUbFdsWUZtamFRMEdoNEtIR2gyY3k1U2EwSm5TR3BqZGt0VFJrSlhibmxMZUdKSWRqSlRPVzQK”,“WORKERS”:2,“WORKER_TTL”:30,“WORKING_DIRECTORY”:"/var/lib/pulp/tmp",“WSGI_APPLICATION”:“pulpcore.app.wsgi.application”,“X_FRAME_OPTIONS”:“DENY”,“YEAR_MONTH_FORMAT”:"F Y”,"_DEFAULT_FILE_STORAGE":“pulpcore.app.models.storage.FileSystem”,"_STORAGES":{“default”:{“BACKEND”:“pulpcore.app.models.storage.FileSystem”},“staticfiles”:{“BACKEND”:“django.contrib.staticfiles.storage.StaticFilesStorage”}}}

#15

Thanks!
Before anything, your MEDIA_ROOT is actually empty. Can you try setting it to " /var/lib/pulp/media"?

#16

I will do this.

Just from the very (± 4 yrs back) start this was set to,
MEDIA_ROOT = '' as it was suggested back then.
And still are for our two current running GitHub CI/CDs.

Our lab is down on the moment and as soon it is back, I’ll surely add this.

Thanks a mil.

1 Like