Pulp in one container with external database

Support
#1

Problem:

Hello All

Guidance / best practice required please.

Note: Running pulp in container is working 100% no problems there.

The new goal is now to:

Run pulp3 in one container (quay.io/pulp/pulp) with:

  • An external database
  • Internal database disabled (prevented from startup / being created)
  • Redis disabled

Once the above is working.
Add pulp-worker minimal containers as needed. (quay.io/pulp/pulp-minimal)

I have this and the DB is up:

Will the below entries go to /etc/pulp/settings.py in the container?

DATABASES__default__ENGINE    = "django.db.backends.postgresql"
DATABASES__default__NAME      = "pulp3_sandbox"
DATABASES__default__USER      = "pulpuser"
DATABASES__default__PASSWORD  = "pulp-passwd-xxx"
DATABASES__default__HOST      = "pghst13.abcgroup.net"
DATABASES__default__PORT      = "50423"
DB_ENCRYPTION_KEY             = "/local/database_fields.symmetric.key"
CACHE_ENABLED                 = "False"

Expected outcome:

Using pulp/pulp container with external database - internal not created/used

Pulpcore version:

pulpcore 3.27.0

Pulp plugins installed and their versions:

Version in image:
pulp-ansible 0.18.0
pulp-certguard 1.6.5
pulp-container 2.15.0
pulp-deb 2.21.0
pulp-file 1.14.3
pulp-glue 0.19.2
pulp-maven 0.6.0
pulp-ostree 2.1.0
pulp-python 3.10.0
pulp-rpm 3.21.0

Operating system - distribution and version:

SLES15 SP4

Thank you,
Jan

2 Likes
#2

The format if configured in python is a little bit different. The versions you have should work as env vars though. All the possible ways to specify settings are documented with dynaconf.

#3

Thanks so much.

I have tried and the external is up but clashes with the internal PostgreSQL

How to disable the internal PostgresSQL?

If the ā€˜how-toā€™ information is available please mention the URL
I have missed it then.
.
As user ā€˜pulpā€™ in the container:
env|sort

PULP_API_WORKERS=2
PULP_CACHE_ENABLED=False
PULP_CONTENT_WORKERS=2
PULP_DATABASES__default__ENGINE=django.db.backends.postgresql
PULP_DATABASES__default__HOST=pghst13.abcgroup.net
PULP_DATABASES__default__NAME=pulp3_sandbox
PULP_DATABASES__default__PASSWORD=pulp-passwd-xxx
PULP_DATABASES__default__PORT=50423
PULP_DATABASES__default__USER=pulpuser
PULP_DB_ENCRYPTION_KEY=/local/database_fields.symmetric.key
PULP_GUNICORN_RELOAD=false
PULP_GUNICORN_TIMEOUT=90
PULP_HTTPS=true
PULP_OTEL_ENABLED=false
PULP_SETTINGS=/etc/pulp/settings.py
PULP_STATIC_ROOT=/var/lib/operator/static/
PULP_WORKERS=1
PWD=/
PYTHONUNBUFFERED=0
S6_BEHAVIOUR_IF_STAGE2_FAILS=1
S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0

bash-4.4$ cat /etc/pulp/settings.py

WORKERS                       = "1"
CONTENT_WORKERS               = "2"
CONTENT_ORIGIN                = "https://10.1.10.101:30443"
ANSIBLE_API_HOSTNAME          = "https://10.1.10.101:30443"
ANSIBLE_CONTENT_HOSTNAME      = "https://10.1.10.101:30443/pulp/content"
TOKEN_AUTH_DISABLED           = "True"
REDIRECT_TO_OBJECT_STORAGE    = "False"
DEFAULT_FILE_STORAGE          = "storages.backends.s3boto3.S3Boto3Storage"
AWS_S3_ENDPOINT_URL           = "https://v50324s1.abcgroup.net:9099"
AWS_STORAGE_BUCKET_NAME       = "c000009"
AWS_ACCESS_KEY_ID             = "vc000009_abcu1@europe.abc.corp"
AWS_SECRET_ACCESS_KEY         = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
AWS_S3_ADDRESSING_STYLE       = "path"
AWS_DEFAULT_ACL               = "None"
AWS_S3_VERIFY                 = "/local/abc_pki.pem"
DATABASES__default__ENGINE    = "django.db.backends.postgresql_psycopg2"
DATABASES__default__NAME      = "pulp3_sandbox"
DATABASES__default__USER      = "pulpuser"
DATABASES__default__PASSWORD  = "6yyyyxxxxxxxxx"
DATABASES__default__HOST      = "pghst13.abcgroup.net"
DATABASES__default__PORT      = "50423"
CACHE_ENABLED                 = "False"
DB_ENCRYPTION_KEY             = "/local/database_fields.symmetric.key"

The container log and the container is never usable as it never gets out of this loop.

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
fix-attrs: info: applying /etc/fix-attrs.d/postgres
s6-chown: fatal: unable to chown /var/lib/pgsql/backups
data: No such file or directory
fix-attrs: warning: fix-attrs is deprecated, please fix volume permissions in your container manager instead
fix-attrs: warning: some fix files failed to apply
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service postgres-init: starting
s6-rc: info: service fix-pulp-perms: starting
s6-rc: info: service db-fields-key-create: starting
s6-rc: info: service redis: starting
Calling /etc/init/postgres-init
Calling /etc/init/db-fields-key-create
s6-rc: info: service redis successfully started
Calling /etc/init/fix-pulp-perms
[oneshot] db-fields-key-create: creating database fields key
Checking /var/lib/pulp/media permissions
Checking /var/lib/pulp/scripts permissions
s6-rc: info: service db-fields-key-create successfully started
s6-rc: info: service certs: starting
Calling /etc/init/certs
Checking /var/lib/pulp/tmp permissions
[oneshot] certs: generating token key pair
s6-rc: info: service fix-pulp-perms successfully started
[oneshot] postgres-init: initdb -E UTF8 --locale=C.UTF-8 --pgdata /var/lib/pgsql/data
read EC key
writing EC key
[oneshot] certs: finished generating token key pair
s6-rc: info: service certs successfully started
The files belonging to this database system will be owned by user ā€œpostgresā€.
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/pgsql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/pgsql/data -l logfile start

s6-rc: info: service postgres-init successfully started
s6-rc: info: service postgresql: starting
s6-rc: info: service postgresql successfully started
s6-rc: info: service postgres-prepare: starting
Calling /etc/init/postgres-prepare
2023-06-01 15:44:10.291 CEST [148] LOG:  redirecting log output to logging collector process
2023-06-01 15:44:10.291 CEST [148] HINT:  Future log output will appear in directory "log".
[oneshot] postgres-prepare: createuser pulp
[oneshot] postgres-prepare: createdb --encoding=utf-8 --locale=en_US.UTF-8 -T template0 -O pulp pulp
[oneshot] postgres-prepare: running Pulp migrations
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, maven, ostree, python, rpm, sessions
Running migrations:
  No migrations to apply.
s6-rc: info: service postgres-prepare successfully started
s6-rc: info: service add-workers: starting
s6-rc: info: service pulpcore-worker@1: starting
s6-rc: info: service pulpcore-content: starting
s6-rc: info: service pulpcore-api: starting
s6-rc: info: service pulpcore-worker@1 successfully started
s6-rc: info: service pulpcore-content successfully started
s6-rc: info: service pulpcore-api successfully started
s6-rc: info: service nginx: starting
Calling /etc/init/add-workers
s6-rc: info: service nginx successfully started
Checking for database migrations
Checking for database migrations
Calling /etc/init/nginx
Checking for database migrations
[oneshot] add-workers: no workers to add
s6-rc: info: service add-workers successfully started
s6-rc: info: service legacy-services: starting
'/nginx/ssl_nginx.conf' -> '/etc/nginx/nginx.conf'
s6-rc: info: service legacy-services successfully started
 [ ] 0001_initial
 [ ] 0002_advanced_collections
 [ ] 0003_add_tags_and_collectionversion_fields
 [ ] 0004_add_fulltext_search_indexes
 [ ] 0005_collectionversion_is_highest
 [ ] 0006_remove_whitelist_and_alter_collection_version_name
 [ ] 0007_collectionversion_is_certified
 [ ] 0008_collectionremote_requirements_file
 [ ] 0009_collectionimport
 [ ] 0010_ansible_related_names
 [ ] 0011_collectionimport
 [ ] 0012_auto_20190906_2253
 [ ] 0013_pulp_fields
 [ ] 0014_certification_enum
 [ ] 0015_ansiblerepository
 [ ] 0016_add_extension
 [ ] 0017_increase_length_collectionversion_fields
 [ ] 0018_fix_collection_relative_path
 [ ] 0019_collection_token
 [ ] 0020_auto_20200810_1926
 [ ] 0021_rename_role_remote
 [ ] 0022_URLField_to_CharField
 [ ] 0023_alter_requirements_file_field
 [ ] 0024_remove_collectionversion_certification
 [ ] 0025_increase_collection_version_version_size
 [ ] 0026_deprecation_per_repository
 [ ] 0027_tag_length
 [ ] 0028_collectionversion_namespace_length
 [ ] 0029_manifest_and_files_json_fields
 [ ] 0030_collectionversion_requires_ansible
 [ ] 0031_ansiblerepository_last_synced_metadata_time
 [ ] 0032_collectionremote_sync_dependencies
 [ ] 0033_swap_distribution_model
 [ ] 0034_handle_jsonfield_warnings
 [ ] 0035_deprecation_content
 [ ] 0036_update_repository_content
 [ ] 0037_gitremote
 [ ] 0038_collectionversionsignature
 [ ] 0039_collectionremote_signed_only
 [ ] 0040_ansiblerepository_keyring
 [ ] 0041_alter_collectionversion_collection
 [ ] 0042_ansiblerepository_gpgkey
 [ ] 0043_alter_collectionversionsignature_data
 [ ] 0044_alter_collectionremote_token
 [ ] 0045_downloadlog
 [ ] 0046_add_fulltext_search_fix
 [ ] 0047_ansible_namespace
 [ ] 0048_collectionversionmark
 [ ] 0049_rbac_permissions
 [ ] 0050_crossrepositorycollectionversionindex
 [ ] 0051_cvindex_build
 [ ] 0052_alter_ansiblecollectiondeprecated_content_ptr_and_more
 [ ] 0001_initial
 [ ] 0002_alter_permission_name_max_length
 [ ] 0003_alter_user_email_max_length
 [ ] 0004_alter_user_username_opts
 [ ] 0005_alter_user_last_login_null
 [ ] 0006_require_contenttypes_0002
 [ ] 0007_alter_validators_add_error_messages
 [ ] 0008_alter_user_username_max_length
 [ ] 0009_alter_user_last_name_max_length
 [ ] 0010_alter_group_name_max_length
 [ ] 0011_update_proxy_permissions
 [ ] 0012_alter_user_first_name_max_length
 [ ] 0001_initial
 [ ] 0002_alter_rhsmcertguard_contentguard_ptr_and_more
 [ ] 0001_initial
 [ ] 0002_containerrepository
 [ ] 0003_oci_mediatype
 [ ] 0004_upload
 [ ] 0005_contentredirectcontentguard
 [ ] 0006_containerpushrepository
 [ ] 0007_clear_tags_artifacts_refs
 [ ] 0008_include_exclude_tags
 [ ] 0009_container_namespace
 [ ] 0010_remove_uploadchunk
 [ ] 0011_add_container_repository_permissions
 [ ] 0012_add_container_namespace_permissions
 [ ] 0013_add_pull_push_permissions
 [ ] 0014_containerdistribution_private
 [ ] 0015_manage_tags_push_repo
 [ ] 0016_add_delete_versions_permission
 [ ] 0017_add_granular_perms
 [ ] 0018_containerdistribution_description
 [ ] 0019_DATA_distribution_model_swap
 [ ] 0020_update_push_repo_perms
 [ ] 0021_data_move_redirect_content_guard_to_core
 [ ] 0022_delete_contentredirectcontentguard
 [ ] 0023_manifestsignature
 [ ] 0024_containerremote_sigstore
 [ ] 0025_signature_stored_in_textfield
 [ ] 0026_manifest_signing_service
 [ ] 0027_data_translate_perms_to_roles
 [ ] 0028_add_role_manage_permissions
 [ ] 0029_remove_blob_media_type
 [ ] 0030_enforce_tagged_manifest_reference
 [ ] 0031_replace_charf_with_textf
 [ ] 0032_upload_artifact
 [ ] 0033_raise_warning_for_repair
 [ ] 0034_translate_signed_schema
 [ ] 0035_alter_blob_content_ptr_and_more
 [ ] 0036_containerpushrepository_pending_blobs_manifests
 [ ] 0001_initial
 [ ] 0002_remove_content_type_name
 [ ] 0001_initial
 [ ] 0002_increase_artifact_size_field
 [ ] 0003_remove_upload_completed
 [ ] 0004_add_duplicated_reserved_resources
 [ ] 0005_progressreport_code
 [ ] 0006_repository_plugin_managed
 [ ] 0007_delete_progress_proxies
 [ ] 0008_published_metadata_as_content
 [ ] 0009_remove_task_non_fatal_errors
 [ ] 0010_pulp_fields
 [ ] 0011_relative_path
 [ ] 0012_auto_20191104_2000
 [ ] 0013_repository_pulp_type
 [ ] 0014_remove_repository_plugin_managed
 [ ] 0015_auto_20191112_1426
 [ ] 0016_charfield_to_textfield
 [ ] 0017_remove_task_parent
 [ ] 0018_auto_20191127_2350
 [ ] 0019_add_signing_service_model
 [ ] 0020_change_publishedartifact_constraints
 [ ] 0021_add_signing_service_foreign_key
 [ ] 0022_rename_last_version
 [ ] 0023_change_exporter_models
 [ ] 0024_use_local_storage_for_uploads
 [ ] 0025_task_parent_task
 [ ] 0026_task_group
 [ ] 0027_export_backend
 [ ] 0028_import_importer_pulpimporter_pulpimporterrepository
 [ ] 0029_export_delete
 [ ] 0030_taskgroup_all_tasks_dispatched
 [ ] 0031_import_export_validate_params
 [ ] 0032_export_to_chunks
 [ ] 0033_increase_remote_artifact_size_field
 [ ] 0034_groupprogressreport
 [ ] 0035_content_upstream_id
 [ ] 0036_unprotect_last_export
 [ ] 0037_pulptemporaryfile
 [ ] 0038_repository_remote
 [ ] 0039_change_download_concurrency
 [ ] 0040_set_admin_is_staff
 [ ] 0041_accesspolicy
 [ ] 0042_rbac_for_tasks
 [ ] 0043_toc_attribute
 [ ] 0044_temp_file_artifact_field
 [ ] 0045_accesspolicy_permissions_allow_null
 [ ] 0046_task__resource_job_id
 [ ] 0047_improve_orphan_cleanup
 [ ] 0048_fips_checksums
 [ ] 0049_add_file_field_to_uploadchunk
 [ ] 0050_namespace_access_policies
 [ ] 0051_timeoutfields
 [ ] 0052_tasking_logging_cid
 [ ] 0053_remote_headers
 [ ] 0054_add_public_key
 [ ] 0055_label
 [ ] 0056_remote_rate_limit
 [ ] 0057_add_label_indexes
 [ ] 0058_accesspolicy_customized
 [ ] 0059_proxy_creds
 [ ] 0060_data_migration_proxy_creds
 [ ] 0061_call_handle_artifact_checksums_command
 [ ] 0062_add_new_distribution_mastermodel
 [ ] 0063_repository_retained_versions
 [ ] 0064_repository_user_hidden
 [ ] 0064_add_new_style_task_columns
 [ ] 0065_merge_20210615_1211
 [ ] 0066_download_concurrency_and_retry_changes
 [ ] 0067_add_protect_to_task_reservation
 [ ] 0068_add_timestamp_of_interest
 [ ] 0069_update_json_fields
 [ ] 0070_rename_retained_versions
 [ ] 0071_filesystemexport_filesystemexporter
 [ ] 0072_add_method_to_filesystem_exporter
 [ ] 0073_encrypt_remote_fields
 [ ] 0074_acs
 [ ] 0075_rbaccontentguard
 [ ] 0076_remove_reserved_resource
 [ ] 0077_move_remote_url_credentials
 [ ] 0078_grouprole_role_userrole
 [ ] 0079_rename_permissions_assignment_accesspolicy_creation_hooks
 [ ] 0080_proxy_group_model
 [ ] 0081_reapplabel_group_permissions
 [ ] 0082_add_manage_roles_permissions
 [ ] 0083_alter_group_options
 [ ] 0084_alter_rbaccontentguard_options
 [ ] 0085_contentredirectcontentguard
 [ ] 0086_task_json_fields
 [ ] 0087_taskschedule
 [ ] 0088_accesspolicy_queryset_scoping
 [ ] 0089_alter_contentredirectcontentguard_options
 [ ] 0090_char_to_text_field
 [ ] 0091_systemid
 [ ] 0092_alter_upload_options
 [ ] 0093_add_info_field_repositoryversion
 [ ] 0094_protect_repository_content
 [ ] 0095_artifactdistribution
 [ ] 0096_alter_task_logging_cid
 [ ] 0097_remove_telemetry_task_schedule
 [ ] 0098_pulp_labels
 [ ] 0099_versions_field
 [ ] 0100_upstreampulp
 [ ] 0101_add_domain
 [ ] 0102_add_domain_relations
 [ ] 0103_alter_export_task
 [ ] 0104_delete_label
 [ ] 0105_abstract_uuid_gen
 [ ] 0106_alter_artifactdistribution_distribution_ptr_and_more
 [ ] 0107_distribution_hidden
 [ ] 0001_initial
 [ ] 0002_auto_20190905_1000
 [ ] 0003_rename_package_field
 [ ] 0004_auto_20191014_1722
 [ ] 0005_rename_release_file
 [ ] 0006_debrepository
 [ ] 0007_create_metadata_models
 [ ] 0008_debremote_gpgkey
 [ ] 0009_apt_release_signing_service
 [ ] 0010_debpublication_signing_service
 [ ] 0011_rename_models_to_apt
 [ ] 0012_auto_20200803_1337
 [ ] 0013_aptremote_ignore_missing_package_indices
 [ ] 0014_swap_distribution_model
 [ ] 0015_add_custom_fields_to_packages
 [ ] 0016_switch_to_newer_jsonfield_model
 [ ] 0017_allow_longer_string_lists
 [ ] 0018_textfield_conversion
 [ ] 0019_immutable_metadata_constraints
 [ ] 0020_remove_fk_relations_in_packageindex_and_installerfileindex
 [ ] 0021_remove_release_from_structure_types
 [ ] 0022_alter_aptdistribution_distribution_ptr_and_more
 [ ] 0001_initial
 [  [ ] 0001_initial
 [ ] 0002_advanced_collections
 [ ] 0003_add_tags_and_collectionversion_fields
 [ ] 0004_add_fulltext_search_indexes
 [ ] 0005_collectionversion_is_highest
 [ ] 0006_remove_whitelist_and_alter_collection_version_name
 [ ] 0007_collectionversion_is_certified
 [ ] 0008_collectionremote_requirements_file
 [ ] 0009_collectionimport
 [ ] 0010_ansible_related_names
 [ ] 0011_collectionimport
 [ ] 0012_auto_20190906_2253
 [ ] 0013_pulp_fields
 [ ] 0014_certification_enum
 [ ] 0015_ansiblerepository
 [ ] 0016_add_extension
 [ ] 0017_increase_length_collectionversion_fields
 [ ] 0018_fix_collection_relative_path
 [ ] 0019_collection_token
 [ ] 0020_auto_20200810_1926
 [ ] 0021_rename_role_remote
 [ ] 0022_URLField_to_CharField
 [ ] 0023_alter_requirements_file_field
 [ ] 0024_remove_collectionversion_certification
 [ ] 0025_increase_collection_version_version_size
 [ ] 0026_deprecation_per_repository
 [ ] 0027_tag_length
 [ ] 0028_collectionversion_namespace_length
 [ ] 0029_manifest_and_files_json_fields
 [ ] 0030_collectionversion_requires_ansible
 [ ] 0031_ansiblerepository_last_synced_metadata_time
 [ ] 0032_collectionremote_sync_dependencies
 [ ] 0033_swap_distribution_model
 [ ] 0034_handle_jsonfield_warnings
 [ ] 0035_deprecation_content
 [ ] 0036_update_repository_content
 [ ] 0037_gitremote
 [ ] 0038_collectionversionsignature
 [ ] 0039_collectionremote_signed_only
 [ ] 0040_ansiblerepository_keyring
 [ ] 0041_alter_collectionversion_collection
 [ ] 0042_ansiblerepository_gpgkey
 [ ] 0043_alter_collectionversionsignature_data
 [ ] 0044_alter_collectionremote_token
 [ ] 0045_downloadlog
 [ ] 0046_add_fulltext_search_fix
 [ ] 0047_ansible_namespace
 [ ] 0048_collectionversionmark
 [ ] 0049_rbac_permissions
 [ ] 0050_crossrepositorycollectionversionindex
 [ ] 0051_cvindex_build
 [ ] 0052_alter_ansiblecollectiondeprecated_content_ptr_and_more
 [ ] 0001_initial
 [ ] 0002_alter_permission_name_max_length
 [ ] 0003_alter_user_email_max_length
 [ ] 0004_alter_user_username_opts
 [ ] 0005_alter_user_last_login_null
 [ ] 0006_require_contenttypes_0002
 [ ] 0007_alter_validators_add_error_messages
 [ ] 0008_alter_user_username_max_length
 [ ] 0009_alter_user_last_name_max_length
 [ ] 0010_alter_group_name_max_length
 [ ] 0011_update_proxy_permissions
 [ ] 0012_alter_user_first_name_max_length
 [ ] 0001_initial
 [ ] 0002_alter_rhsmcertguard_contentguard_ptr_and_more
 [ ] 0001_initial
 [ ] 0002_containerrepository
 [ ] 0003_oci_mediatype
 [ ] 0004_upload
 [ ] 0005_contentredirectcontentguard
 [ ] 0006_containerpushrepository
 [ ] 0007_clear_tags_artifacts_refs
 [ ] 0008_include_exclude_tags
 [ ] 0009_container_namespace
 [ ] 0010_remove_uploadchunk
 [ ] 0011_add_container_repository_permissions
 [ ] 0012_add_container_namespace_permissions
 [ ] 0013_add_pull_push_permissions
 [ ] 0014_containerdistribution_private
 [ ] 0015_manage_tags_push_repo
 [ ] 0016_add_delete_versions_permission
 [ ] 0017_add_granular_perms
 [ ] 0018_containerdistribution_description
 [ ] 0019_DATA_distribution_model_swap
 [ ] 0020_update_push_repo_perms
 [ ] 0021_data_move_redirect_content_guard_to_core
 [ ] 0022_delete_contentredirectcontentguard
 [ ] 0023_manifestsignature
 [ ] 0024_containerremote_sigstore
 [ ] 0025_signature_stored_in_textfield
 [ ] 0026_manifest_signing_service
 [ ] 0027_data_translate_perms_to_roles
 [ ] 0028_add_role_manage_permissions
 [ ] 0029_remove_blob_media_type
 [ ] 0030_enforce_tagged_manifest_reference
 [ ] 0031_replace_charf_with_textf
 [ ] 0032_upload_artifact
 [ ] 0033_raise_warning_for_repair
 [ ] 0034_translate_signed_schema
 [ ] 0035_alter_blob_content_ptr_and_more
 [ ] 0036_containerpushrepository_pending_blobs_manifests
 [ ] 0001_initial
 [ ] 0002_remove_content_type_name
 [ ] 0001_initial
 [ ] 0002_increase_artifact_size_field
 [ ] 0003_remove_upload_completed
 [ ] 0004_add_duplicated_reserved_resources
 [ ] 0005_progressreport_code
 [ ] 0006_repository_plugin_managed
 [ ] 0007_delete_progress_proxies
 [ ] 0008_published_metadata_as_content
 [ ] 0009_remove_task_non_fatal_errors
 [ ] 0010_pulp_fields
 [ ] 0011_relative_path
 [ ] 0012_auto_20191104_2000
 [ ] 0013_repository_pulp_type
 [ ] 0014_remove_repository_plugin_managed
 [ ] 0015_auto_20191112_1426
 [ ] 0016_charfield_to_textfield
 [ ] 0017_remove_task_parent
 [ ] 0018_auto_20191127_2350
 [ ] 0019_add_signing_service_model
 [ ] 0020_change_publishedartifact_constraints
 [ ] 0021_add_signing_service_foreign_key
 [ ] 0022_rename_last_version
 [ ] 0023_change_exporter_models
 [ ] 0024_use_local_storage_for_uploads
 [ ] 0025_task_parent_task
 [ ] 0026_task_group
 [ ] 0027_export_backend
 [ ] 0028_import_importer_pulpimporter_pulpimporterrepository
 [ ] 0029_export_delete
 [ ] 0030_taskgroup_all_tasks_dispatched
 [ ] 0031_import_export_validate_params
 [ ] 0032_export_to_chunks
 [ ] 0033_increase_remote_artifact_size_field
 [ ] 0034_groupprogressreport
 [ ] 0035_content_upstream_id
 [ ] 0036_unprotect_last_export
 [ ] 0037_pulptemporaryfile
 [ ] 0038_repository_remote
 [ ] 0039_change_download_concurrency
 [ ] 0040_set_admin_is_staff
 [ ] 0041_accesspolicy
 [ ] 0042_rbac_for_tasks
 [ ] 0043_toc_attribute
 [ ] 0044_temp_file_artifact_field
 [ ] 0045_accesspolicy_permissions_allow_null
 [ ] 0046_task__resource_job_id
 [ ] 0047_improve_orphan_cleanup
 [ ] 0048_fips_checksums
 [ ] 0049_add_file_field_to_uploadchunk
 [ ] 0050_namespace_access_policies
 [ ] 0051_timeoutfields
 [ ] 0052_tasking_logging_cid
 [ ] 0053_remote_headers
 [ ] 0054_add_public_key
 [ ] 0055_label
 [ ] 0056_remote_rate_limit
 [ ] 0057_add_label_indexes
 [ ] 0058_accesspolicy_customized
 [ ] 0059_proxy_creds
 [ ] 0060_data_migration_proxy_creds
 [ ] 0061_call_handle_artifact_checksums_command
 [ ] 0062_add_new_distribution_mastermodel
 [ ] 0063_repository_retained_versions
 [ ] 0064_repository_user_hidden
 [ ] 0064_add_new_style_task_columns
 [ ] 0065_merge_20210615_1211
 [ ] 0066_download_concurrency_and_retry_changes
 [ ] 0067_add_protect_to_task_reservation
 [ ] 0068_add_timestamp_of_interest
 [ ] 0069_update_json_fields
 [ ] 0070_rename_retained_versions
 [ ] 0071_filesystemexport_filesystemexporter
 [ ] 0072_add_method_to_filesystem_exporter
 [ ] 0073_encrypt_remote_fields
 [ ] 0074_acs
 [ ] 0075_rbaccontentguard
 [ ] 0076_remove_reserved_resource
 [ ] 0077_move_remote_url_credentials
 [ ] 0078_grouprole_role_userrole
 [ ] 0079_rename_permissions_assignment_accesspolicy_creation_hooks
 [ ] 0080_proxy_group_model
 [ ] 0081_reapplabel_group_permissions
 [ ] 0082_add_manage_roles_permissions
 [ ] 0083_alter_group_options
 [ ] 0084_alter_rbaccontentguard_options
 [ ] 0085_contentredirectcontentguard
 [ ] 0086_task_json_fields
 [ ] 0087_taskschedule
 [ ] 0088_accesspolicy_queryset_scoping
 [ ] 0089_alter_contentredirectcontentguard_options
 [ ] 0090_char_to_text_field
 [ ] 0091_systemid
 [ ] 0092_alter_upload_options
 [ ] 0093_add_info_field_repositoryversion
 [ ] 0094_protect_repository_content
 [ ] 0095_artifactdistribution
 [ ] 0096_alter_task_logging_cid
 [ ] 0097_remove_telemetry_task_schedule
 [ ] 0098_pulp_labels
 [ ] 0099_versions_field
 [ ] 0100_upstreampulp
 [ ] 0101_add_domain
 [ ] 0102_add_domain_relations
 [ ] 0103_alter_export_task
 [ ] 0104_delete_label
 [ ] 0105_abstract_uuid_gen
 [ ] 0106_alter_artifactdistribution_distribution_ptr_and_more
 [ ] 0107_distribution_hidden
 [ ] 0001_initial
 [ ] 0002_auto_20190905_1000
 [ ] 0003_rename_package_field
 [ ] 0004_auto_20191014_1722
 [ ] 0005_rename_release_file
 [ ] 0006_debrepository
 [ ] 0007_create_metadata_models
 [ ] 0008_debremote_gpgkey
 [ ] 0009_apt_release_signing_service
 [ ] 0010_debpublication_signing_service
 [ ] 0011_rename_models_to_apt
 [ ] 0012_auto_20200803_1337
 [ ] 0013_aptremote_ignore_missing_package_indices
 [ ] 0014_swap_distribution_model
 [ ] 0015_add_custom_fields_to_packages
 [ ] 0016_switch_to_newer_jsonfield_model
 [ ] 0017_allow_longer_string_lists
 [ ] 0018_textfield_conversion
 [ ] 0019_immutable_metadata_constraints
 [ ] 0020_remove_fk_relations_in_packageindex_and_installerfileindex
 [ ] 0021_remove_release_from_structure_types
 [ ] 0022_alter_aptdistribution_distribution_ptr_and_more
 [ ] 0001_initial
 [ ] 0002_file_related_names
 [ ] 0003_auto_20191014_1721
 [ ] 0004_filefilesystemexporter
 [ ] 0005_filerepository
 [ ] 0006_delete_filefilesystemexporter
 [ ] 0007_filefilesystemexporter
 [ ] 0008_add_manifest_field
 [ ] 0009_move_data_to_new_master_distribution_model
 [ ] 0010_auto_publish
 [ ] 0011_fix_auto_publish
 [ ] 0012_delete_filefilesystemexporter
 [ ] 0013_file_acs
 [ ] 0014_new_rbac_permissions
 [ ] 0015_allow_null_manifest
 [ ] 0016_add_domain
 [ ] 0017_alter_filealternatecontentsource_alternatecontentsource_ptr_and_more
 [ ] 0001_initial
 [ ] 0002_maven_related_names
 [ ] 0003_mavenrepository
 [ ] 0004_swap_distribution_models
 [ ] 0005_mavenmetadata
 [ ] 0006_alter_mavenartifact_content_ptr_and_more
 [ ] 0001_initial
 [ ] 0002_add_relative_path_uniqueness
 [ ] 0003_create_many_to_many_objs_commits
 [ ] 0004_add_include_exclude_refs
 [ ] 0005_add_static_delta_support
 [ ] 0006_alter_pointers_to_related_models_globally
 [ ] 0001_initial
 [ ] 0002_pythonpackagecontent_python_version
 [ ] 0003_new_sync_filters
 [ ] 0004_DATA_swap_distribution_model
 [ ] 0005_pythonpackagecontent_sha256
 [ ] 0006_pythonrepository_autopublish
 [ ] 0007_pythonpackagecontent_mv-2-1
 [ ] 0008_pythonpackagecontent_unique_sha256
 [ ] 0009_pythondistribution_allow_uploads
 [ ] 0010_update_json_field
 [ ] 0011_alter_pythondistribution_distribution_ptr_and_more
 [ ] 0001_initial
 [ ] 0002_updaterecord_reboot_suggested
 [ ] 0003_DATA_incorrect_json
 [ ] 0004_add_metadata_signing_service_fk
 [ ] 0005_optimize_sync
 [ ] 0006_opensuse_support
 [ ] 0007_checksum_types
 [ ] 0008_advisory_pkg_sumtype_as_int
 [ ] 0009_revision_null
 [ ] 0010_revision_null_redo
 [ ] 0011_rpmremote_sles_auth_token
 [ ] 0012_remove_pkg_group_env_cat_related_pkgs
 [ ] 0013_RAW_rpm_evr_extension
 [ ] 0014_rpmrepository_package_retention_policy
 [ ] 0015_repo_metadata
 [ ] 0016_dist_tree_nofk
 [ ] 0017_merge_advisory_collections
 [ ] 0018_updatecollection__update_record
 [ ] 0019_migrate_updatecollection_data
 [ ] 0020_remove_updatecollection_m2m
 [ ] 0021_rename_updatecollection_update_record
 [ ] 0022_add_collections_related_name
 [ ] 0023_increase_distribution_release_short
 [ ] 0024_change_subrepo_relation_properties
 [ ] 0025_remove_orphaned_subrepos
 [ ] 0026_add_gpgcheck_options
 [ ] 0027_checksum_null
 [ ] 0028_rpmrepository_last_sync_repomd_cheksum
 [ ] 0029_rpmpublication_sqlite_metadata
 [ ] 0030_DATA_fix_updaterecord
 [ ] 0031_modulemd_static_context
 [ ] 0032_ulnremote
 [ ] 0033_new_distribution_model
 [ ] 0034_auto_publish
 [ ] 0035_fix_auto_publish
 [ ] 0036_checksum_type
 [ ] 0037_DATA_remove_rpmrepository_sub_repo
 [ ] 0037_update_json_field
 [ ] 0038_fix_sync_optimization
 [ ] 0039_disttree_digest
 [ ] 0040_rpmalternatecontentsource
 [ ] 0041_modulemdobsolete
 [ ] 0042_alter_repometadatafile_data_type
 [ ] 0043_textfield_conversion
 [ ] 0044_noartifact_modules
 [ ] 0045_modulemd_fields
 [ ] 0046_rbac_perms
 [ ] 0047_modulemd_datefield
 [ ] 0048_artifacts_dependencies_fix
 [ ] 0049_profiles_fix
 [ ] 0050_alter_addon_pulp_id_alter_checksum_pulp_id_and_more
 [ ] 0001_initial
Database migrated!
ready
Calling /etc/init/pulpcore-worker

.
. repeat omittedā€¦

 [ ] 0044_noartifact_modules
 [ ] 0045_modulemd_fields
 [ ] 0046_rbac_perms
 [ ] 0047_modulemd_datefield
 [ ] 0048_artifacts_dependencies_fix
 [ ] 0049_profiles_fix
 [ ] 0050_alter_addon_pulp_id_alter_checksum_pulp_id_and_more
 [ ] 0001_initial
pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
pulp [None]: pulpcore.tasking.pulpcore_worker:INFO: New worker '194@f700517cd9c3' discovered
 [ ] 0001_initial
 [ ] 0002_advanced_collections
 [ ] 0003_add_tags_and_collectionversion_fields
 [ ] 0004_add_fulltext_search_indexes

So it is repeating in a loop - 0001 to 0050

Regars

#4

Hi Janr,

If this syntax works, great. I didnā€™t know about it.

Our example of the syntax for the database in settings.py is:
https://docs.pulpproject.org/pulp_oci_images/migration-pulp-installer-to-multi-process/#configure-pulp-to-talk-to-the-database-fedora-36-and-ubuntu-2204-only

Also, I am trying to understand what is the looped output is exactly. Is it the entire output you listed, both in the code block and outside of the code block? Or is it steps 0001 through 0050 of the migration specifically? Because if so, that would be behavior that Iā€™ve never seen before from the migration command:

1 Like
#5

Hello mikedep33

Thank you for the reply, and apologies for the delay

The error we see is expected as there are now 2 PG DBs referenced.
The free standing one as described in the settings.py and the internal one in /var/lib/pgsql/data/

It is illegal.

My query is, can we use an external PG DB with image pulp/pulp.
That imply to ā€˜disableā€™ the ā€˜internalā€™ DB.

Looking at the PULP HA setup as part of the PULP_API is this

  • pulp in one and a database-config
  • with the DB external.

Thanks,
Jan

1 Like
#6

I have encountered this issue and I wonder if a solution was ever discovered.

I believe itā€™s because the container fires up from fresh, initializes itā€™s local pgsql database, but all the django configs are pointed at an external database. So perhaps something as part of the first time initialization never completes properly.
I notice if I restart the pulp container alone, it appears to skip the first time initialization and then it runs normally. Perhaps it sees the local database files instead of a an empty /var/lib/pgsql/.

I can reproduce this with docker compose where I have a psql container and the pulp multi-process container. This is the relevant part of the settings.py file:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "pulp",
        "USER": "pulp",
        "PASSWORD": "'pulp",
        "HOST": "db",
        "PORT": "5432",
    }
}

where ā€œdbā€ is the name of the pgsql container in the docker compose file.

Btw, Iā€™m hoping to use this in AWS ECS with a separate postgresql database. I do encounter the same issue there as well. As it is reproducible locally with docker compose, there is no need to complicate things by talking about ECS though.

I agree with Janrā€™s thoughts though, it would be nice if we could have the option to disable postgresql as part of the multiprocess container. In our case, it would just be wasted resources.

1 Like