Pulp openshift operator worker nodes stuck in DB migration when db pod is restarted

Support
#1

Hi ,

As part of POC with openshift pulp operator ,When db pod is deleted in a running instance , all other pods are stuck with below

Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…

DB pod says below
“core_contentappstatus”.“name”, “core_contentappstatus”.“last_heartbeat”, “core_contentappstatus”.“versions” FROM “core_contentappstatus” WHERE “core_contentappstatus”.“name” = ‘28@pulp-server-nonprod-content-fd7f88975-xh54b’ LIMIT 21
2024-07-01 11:14:49.016 UTC [6621] ERROR: relation “core_contentappstatus” does not exist at character 242
2024-07-01 11:14:49.016 UTC [6621] STATEMENT: SELECT “core_contentappstatus”.“pulp_id”, “core_contentappstatus”.“pulp_created”, “core_contentappstatus”.“pulp_last_updated”, “core_contentappstatus”.“name”, “core_contentappstatus”.“last_heartbeat”, “core_contentappstatus”.“versions” FROM “core_contentappstatus” WHERE “core_contentappstatus”.“name” = ‘23@pulp-server-nonprod-content-559f4bf567-rz2r8’ LIMIT 21
2024-07-01 11:14:49.624 UTC [6626] ERROR: relation “core_artifact” does not exist at character 28
2024-07-01 11:14:49.624 UTC [6626] STATEMENT: SELECT count(pulp_id) FROM core_artifact WHERE sha224 IS NULL
2024-07-01 11:14:50.759 UTC [6627] ERROR: relation “core_contentappstatus” does not exist at character 242
2024-07-01 11:14:50.759 UTC [6627] STATEMENT: SELECT “core_contentappstatus”.“pulp_id”, “core_contentappstatus”.“pulp_created”, “core_contentappstatus”.“pulp_last_updated”, “core_contentappstatus”.“name”, “core_contentappstatus”.“last_heartbeat”, “core_contentappstatus”.“versions” FROM “core_contentappstatus” WHERE “core_contentappstatus”.“name” = ‘30@pulp-server-nonprod-content-fd7f88975-xh54b’ LIMIT 21
2024-07-01 11:14:51.300 UTC [6628] ERROR: relation “core_artifact” does not exist at character 28
2024-07-01 11:14:51.300 UTC [6628] STATEMENT: SELECT count(pulp_id) FROM core_artifact WHERE sha224 IS NULL
2024-07-01 11:14:53.560 UTC [6631] ERROR: relation “core_contentappstatus” does not exist at character 242
2024-07-01 11:14:53.560 UTC [6631] STATEMENT: SELECT “core_contentappstatus”.“pulp_id”, “core_contentappstatus”.“pulp_created”, “core_contentappstatus”.“pulp_last_updated”, “core_contentappstatus”.“name”, “core_contentappstatus”.“last_heartbeat”, “core_contentappstatus”.“versions” FROM “core_contentappstatus” WHERE “core_contentappstatus”.“name” = ‘22@pulp-server-nonprod-content-559f4bf567-rz2r8’ LIMIT 21
2024-07-01 11:14:55.063 UTC [6632] ERROR: relation “core_contentappstatus” does not exist at character 242
2024-07-01 11:14:55.063 UTC [6632] STATEMENT: SELECT “core_contentappstatus”.“pulp_id”, “core_contentappstatus”.“pulp_created”, “core_contentappstatus”.“pulp_last_updated”, “core_contentappstatus”.“name”, “core_contentappstatus”.“last_heartbeat”, “core_contentappstatus”.“versions” FROM “core_contentappstatus” WHERE “core_contentappstatus”.“name” = ‘28@pulp-server-nonprod-content-fd7f88975-xh54b’ LIMIT 21
2024-07-01 11:14:55.225 UTC [6633] ERROR: relation “core_artifact” does not exist at character 28
2024-07-01 11:14:55.225 UTC [6633] STATEMENT: SELECT count(pulp_id) FROM core_artifact WHERE sha224 IS NULL
2024-07-01 11:14:56.035 UTC [6634] ERROR: relation “core_contentappstatus” does not exist at character 242
2024-07-01 11:14:56.035 UTC [6634] STATEMENT: SELECT “core_contentappstatus”.“pulp_id”, “core_contentappstatus”.“pulp_created”, “core_contentappstatus”.“pulp_last_updated”, “core_contentappstatus”.“name”, “core_contentappstatus”.“last_heartbeat”, “core_contentappstatus”.“versions” FROM “core_contentappstatus” WHERE “core_contentappstatus”.“name” = ‘23@pulp-server-nonprod-content-559f4bf567-rz2r8’ LIMIT 21
2024-07-01 11:14:57.504 UTC [6635] ERROR: relation “core_artifact” does not exist at character 28
2024-07-01 11:14:57.504 UTC [6635] STATEMENT: SELECT count(pulp_id) FROM core_artifact WHERE sha224 IS NULL
2024-07-01 11:14:57.778 UTC [6637] ERROR: relation “core_contentappstatus” does not exist at character 242
2024-07-01 11:14:57.778 UTC [6637] STATEMENT: SELECT “core_contentappstatus”.“pulp_id”, “core_contentappstatus”.“pulp_created”, “core_contentappstatus”.“pulp_last_updated”, “core_contentappstatus”.“name”, “core_contentappstatus”.“last_heartbeat”, “core_contentappstatus”.“versions” FROM “core_contentappstatus” WHERE “core_contentappstatus”.“name” = ‘30@pulp-server-nonprod-content-fd7f88975-xh54b’ LIMIT 21

Any suggestions

#2

Hi @midhuhk,

“[…] When db pod is deleted in a running instance […]”

Was this env PoC deployed with an ephemeral db pod or was the database volume also deleted?
If so, after deleting the postgres pod/volume, all of its data is also lost.
In this case, to recreate the database schema you can run the django migrations through:

  • get the name of a content or api pod, for example:
$ oc get pod -l app=pulp-content -oname
pod/pulp-content-58787cdc8f-p8znk
  • open a remote shell session into the init container and run the migrations:
$ oc rsh -cinit-container pod/pulp-content-58787cdc8f-p8znk
  • run the migration
sh-5.1$ pulpcore-manager migrate
error: Failed to initialize NSS library <----- you can ignore this error message
Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, maven, ostree, python, rpm, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK                                                                                                                                                                                                                                                
  Applying contenttypes.0002_remove_content_type_name... OK                                                                                                                                                                                                                               
  Applying core.0001_initial... OK                                                                                                                                                                                                                                                        
  Applying core.0002_increase_artifact_size_field... OK                                                                                                                                                                                                                                   
  Applying core.0003_remove_upload_completed... OK                                                                                                                                                                                                                                        
  Applying core.0004_add_duplicated_reserved_resources... OK                                                                                                                                                                                                                              
...
Access policy for distributions/file/file created.                                                                                                                                                                                                                                        
Access policy for publications/file/file created.                                                                                                                                                                                                                                         
Access policy for remotes/file/file created.                                                                                                                                                                                                                                              
Access policy for repositories/file/file/versions created.                                                                                                                                                                                                                                
Access policy for repositories/file/file created.

After that, check if all pods are in a Running state again:

$ oc get pods
NAME                                                READY   STATUS    RESTARTS      AGE
pulp-api-84d8fcffbf-lj72p                           1/1     Running   7 (56s ago)   12d
pulp-content-58787cdc8f-p8znk                       1/1     Running   0             15m
pulp-database-0                                     1/1     Running   0             15m
pulp-operator-controller-manager-56d994b744-vjlk9   2/2     Running   2 (11d ago)   12d
pulp-worker-795ddc6569-q4zxx                        1/1     Running   3 (15m ago)   12d
2 Likes
#3

Hi ,pvc is attached to db pod and I see its still mounted in api/content pod .So data is still there I guess .

I cannot rsh to init containers
be3075@dk1lxjump01 ~]$ oc logs pod/pulp-server-nonprod-content-559f4bf567-5rq79
Waiting on postgresql to start…
Postgres started.
Checking for database migrations
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…

[be3075@dk1lxjump01 ~]$ oc rsh -cinit-container pod/pulp-server-nonprod-content-559f4bf567-5rq79
error: unable to upgrade connection: container not found (“init-container”)

Current pod status
[be3075@dk1lxjump01 ~]$ oc get pods
NAME READY STATUS RESTARTS AGE
pulp-operator-controller-manager-699d9c7876-s8zf2 2/2 Running 0 11h
pulp-server-nonprod-api-664b4b8ff5-9grzr 0/1 Init:0/1 0 11h
pulp-server-nonprod-api-664b4b8ff5-rbr5s 0/1 Running 116 (47s ago) 11h
pulp-server-nonprod-content-559f4bf567-5rq79 0/1 Running 0 11h
pulp-server-nonprod-content-559f4bf567-x9hzl 1/1 Running 0 11h
pulp-server-nonprod-database-0 1/1 Running 0 6m44s
pulp-server-nonprod-redis-7448d64656-kcw9m 1/1 Running 0 11h
pulp-server-nonprod-worker-6df9cddfdb-4swrb 1/1 Running 0 11h
pulp-server-nonprod-worker-6df9cddfdb-x9w9v 1/1 Running 1 (6m33s ago) 11h

Can migration executed from pod directly ? Also will data be corrupted/lost with running the migration manually connecting to pods ?

#4

Hum… this is strange.
From your first message, I thought that all of your pods were waiting for db migration:

“When db pod is deleted in a running instance , all other pods are stuck with below […]”

but checking your new message, I can see that there are some in a Running state already (meaning they already checked for pending migrations and started the application process):

pulp-server-nonprod-content-559f4bf567-x9hzl 1/1 Running 0 11h
pulp-server-nonprod-worker-6df9cddfdb-4swrb 1/1 Running 0 11h
pulp-server-nonprod-worker-6df9cddfdb-x9w9v 1/1 Running 1 (6m33s ago) 11h

I guess the ones that are not running yet are the ones you have reprovisioned, right?
But even this way, it is still strange because there is a pod in init state (probably pending migrations):

pulp-server-nonprod-api-664b4b8ff5-9grzr 0/1 Init:0/1 0 11h

and 2 running, but not READY yet (readiness probe failing?):

pulp-server-nonprod-api-664b4b8ff5-rbr5s 0/1 Running 116 (47s ago) 11h
pulp-server-nonprod-content-559f4bf567-5rq79 0/1 Running 0 11h

maybe we have different problems here.
Can you provide a must-gather or adm-inspect outputs, so we can have more information about this?

Also, can you please check if all pulpcore pods (api, content and worker) have the same pulp-minimal image:

oc get pods -ojson| jq -r '.items[]|[.metadata.name,.status.containerStatuses[0].imageID]|@tsv'
#5

Hi ,

Eventhough pods shows as running , log says about wiating for migration.

[be3075@dk1lxjump01 ~]$ oc get pods |grep -i worker
pulp-server-nonprod-worker-6df9cddfdb-4swrb 1/1 Running 0 18h
pulp-server-nonprod-worker-6df9cddfdb-x9w9v 1/1 Running 2 (99m ago) 18h
[be3075@dk1lxjump01 ~]$ oc logs pod/pulp-server-nonprod-worker-6df9cddfdb-4swrb |more
Waiting on postgresql to start…
Postgres started.
Checking for database migrations
Database migration in progress. Waiting…
Database migration in progress. Waiting…

init-contianer checks for /usr/local/bin/pulpcore-manager showmigrations | grep ‘[ ]’ and output is as below
sh-5.1$ /usr/local/bin/pulpcore-manager showmigrations | grep ‘[ ]’
[ ] 0001_initial
[ ] 0002_advanced_collections
[ ] 0003_add_tags_and_collectionversion_fields
[ ] 0004_add_fulltext_search_indexes
[ ] 0005_collectionversion_is_highest
[ ] 0006_remove_whitelist_and_alter_collection_version_name
[ ] 0007_collectionversion_is_certified
[ ] 0008_collectionremote_requirements_file
[ ] 0009_collectionimport
[ ] 0010_ansible_related_names
[ ] 0011_collectionimport
[ ] 0012_auto_20190906_2253
[ ] 0013_pulp_fields
[ ] 0014_certification_enum

[ ] 0045_modulemd_fields
[ ] 0046_rbac_perms
[ ] 0047_modulemd_datefield
[ ] 0048_artifacts_dependencies_fix
[ ] 0049_profiles_fix
[ ] 0050_alter_addon_pulp_id_alter_checksum_pulp_id_and_more
[ ] 0051_alter_distributiontree_unique_together_and_more
[ ] 0052_modulemd_digest
[ ] 0053_rpmdistribution_generate_repo_config
[ ] 0054_remove_gpg_fields
[ ] 0055_add_repo_config_field
[ ] 0056_remove_rpmpublication_sqlite_metadata_and_more
[ ] 0057_rpmpublication_checksum_type_and_more
[ ] 0058_alter_addon_repository_alter_variant_repository
[ ] 0059_rpmpublication_compression_type_and_more
[ ] 0060_rpmpublication_compression_type_empty
[ ] 0061_fix_modulemd_defaults_digest
[ ] 0062_rpmpackagesigningservice_and_more
[ ] 0001_initial

please find the image details:
[be3075@dk1lxjump01 ~]$ oc get pods -ojson| jq -r ‘.items[]|[.metadata.name,.status.containerStatuses[0].imageID]|@tsv’
pulp-operator-controller-manager-699d9c7876-s8zf2 gcr.io/kubebuilder/kube-rbac-proxy@sha256:a8a40ce28a1c769ce905eaa988054bdd3b893079ac294cc36ad7359d63eec3db
pulp-server-nonprod-api-664b4b8ff5-9grzr
pulp-server-nonprod-api-664b4b8ff5-rbr5s quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c
pulp-server-nonprod-content-559f4bf567-5rq79 quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c
pulp-server-nonprod-content-559f4bf567-x9hzl quay.io/pulp/pulp-minimal@sha256:14a792334f74e5e072ca7c4adb184c70afc78c64e3dacbe619a9005ba14a5b16
pulp-server-nonprod-database-0 docker.io/library/postgres@sha256:1386cc5f123a5f69f1ef42a423f8086c59a9001edf7ecca82baf010213691675
pulp-server-nonprod-redis-7448d64656-kcw9m docker.io/library/redis@sha256:e422889e156ebea83856b6ff973bfe0c86bce867d80def228044eeecf925592b
pulp-server-nonprod-worker-6df9cddfdb-4swrb quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c
pulp-server-nonprod-worker-6df9cddfdb-x9w9v quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c

#6

Hi @midhuhk,

There is a pod with a different image version:

pulp-server-nonprod-api-664b4b8ff5-rbr5s quay.io/pulp/pulp-minimal@sha256:6d7d...
pulp-server-nonprod-content-559f4bf567-5rq79 quay.io/pulp/pulp-minimal@sha256:6d7d...
pulp-server-nonprod-content-559f4bf567-x9hzl quay.io/pulp/pulp-minimal@sha256:14a79...   <-----
pulp-server-nonprod-worker-6df9cddfdb-4swrb quay.io/pulp/pulp-minimal@sha256:6d7d...
pulp-server-nonprod-worker-6df9cddfdb-x9w9v quay.io/pulp/pulp-minimal@sha256:6d7d...

this is probably related to a known issue: Workers stuck due to nonexistent migrations · Issue #1228 · pulp/pulp-operator · GitHub
Can you please try to apply the following workaround to see if it fixes the issue:

$ oc edit pulp
...
spec:
  image_pull_policy: Always
...
#7

Image pull policy is already set to Always

[be3075@dk1lxjump01 ~]$ oc get pulp pulp-server-nonprod -o yaml |grep -i pull
image_pull_policy: Always

I have same issue on on another cluster also which has same image version.

[be3075r@yb1334 ~]$ oc get pods -ojson| jq -r ‘.items[]|[.metadata.name,.status.containerStatuses[0].imageID]|@tsv’
pulp-api-5cd964d899-p27nx quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c
pulp-content-67cbc794f8-kd26t quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c
pulp-content-67cbc794f8-vfwt2 quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c
pulp-database-0 docker.io/library/postgres@sha256:3840140f04b26432cd66de0e3a9a63dfc9fed70bb27e161c846b50fe754a9893
pulp-operator-controller-manager-59c78f7cc-x72zq gcr.io/kubebuilder/kube-rbac-proxy@sha256:a8a40ce28a1c769ce905eaa988054bdd3b893079ac294cc36ad7359d63eec3db
pulp-redis-584d45fffb-j7qxs docker.io/library/redis@sha256:01afb31d6d633451d84475ff3eb95f8c48bf0ee59ec9c948b161adb4da882053
pulp-worker-74fcf5dcb9-bdthr quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c
pulp-worker-74fcf5dcb9-djrw4 quay.io/pulp/pulp-minimal@sha256:6d7d6353e5e4ece7dfc9a0697689d34c7fac816c81a89576436b664ccd81a60c

[be3075r@yb1334 ~]$ oc get pods
NAME READY STATUS RESTARTS AGE
pulp-api-5cd964d899-p27nx 0/1 CrashLoopBackOff 13 (4m12s ago) 99m
pulp-content-67cbc794f8-kd26t 1/1 Running 0 81m
pulp-content-67cbc794f8-vfwt2 1/1 Running 0 81m
pulp-database-0 1/1 Running 0 70m
pulp-operator-controller-manager-59c78f7cc-x72zq 2/2 Running 18 (7m44s ago) 6d4h
pulp-redis-584d45fffb-j7qxs 1/1 Running 0 2d6h
pulp-worker-74fcf5dcb9-bdthr 1/1 Running 1 (70m ago) 78m
pulp-worker-74fcf5dcb9-djrw4 1/1 Running 1 (70m ago) 78m

[be3075r@yb1334 ~]$ oc logs pod/pulp-worker-74fcf5dcb9-bdthr |more
Waiting on postgresql to start…
Postgres started.
Checking for database migrations
Database migration in progress. Waiting…
Database migration in progress. Waiting…
Database migration in progress. Waiting…

#8

Ok, in this case, it seems like we need to change our investigation focus to pulp-oci-image (DB Migration fails with ERROR: relation "core_artifact" does not exist at character 28 · Issue #641 · pulp/pulp-oci-images · GitHub) or pulpcore.
I’ll try to reproduce this error in my lab env and will also check with the team if someone can help us with this.

In the meantime, if this error is somehow impacting other tasks in your PoC env, you can try the following workaround:

  • put the operator in an unmanaged state:
oc patch pulp pulp-server-nonprod --type=merge -p '{"spec":{"unmanaged":true}}'
  • remove the init-container and migrations check from pulpcore pods:
oc patch deployment pulp-server-nonprod-api --type=json -p='[{"op": "remove", "path": "/spec/template/spec/initContainers"}]'
oc patch deployment pulp-server-nonprod-api --type=json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["pulpcore-api", "--timeout", "90", "--workers", "2", "--access-logfile", "-"]}]'

oc patch deployment pulp-server-nonprod-content --type=json -p='[{"op": "remove", "path": "/spec/template/spec/initContainers"}]'
oc patch deployment pulp-server-nonprod-content --type=json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["pulpcore-content", "--timeout", "90", "--workers", "2", "--access-logfile", "-"]}]'

oc patch deployment pulp-server-nonprod-worker --type=json -p='[{"op": "remove", "path": "/spec/template/spec/initContainers"}]'
oc patch deployment pulp-server-nonprod-worker --type=json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["pulpcore-worker"]}]'
1 Like
#9

Also, can you please check the output from:

oc debug pod/pulp-<pulpcore pod>
/usr/local/bin/pulpcore-manager showmigrations | grep '\[ \]'

Let’s run the showmigrations command on different pulpcore pods to compare the outputs.

Considering that we found different image versions and the migrations script is stuck, I think this output can be helpful to find what migrations are pending.

#10

sh-5.1$ /usr/local/bin/pulpcore-manager showmigrations | grep ‘[ ]’

[ ] 0001_initial
[ ] 0002_advanced_collections
[ ] 0003_add_tags_and_collectionversion_fields
[ ] 0004_add_fulltext_search_indexes
[ ] 0005_collectionversion_is_highest
[ ] 0006_remove_whitelist_and_alter_collection_version_name
[ ] 0007_collectionversion_is_certified
[ ] 0008_collectionremote_requirements_file
[ ] 0009_collectionimport
[ ] 0010_ansible_related_names
[ ] 0011_collectionimport
[ ] 0012_auto_20190906_2253
[ ] 0013_pulp_fields
[ ] 0014_certification_enum
[ ] 0015_ansiblerepository
[ ] 0016_add_extension
[ ] 0017_increase_length_collectionversion_fields
[ ] 0018_fix_collection_relative_path
[ ] 0019_collection_token
[ ] 0020_auto_20200810_1926
[ ] 0021_rename_role_remote
[ ] 0022_URLField_to_CharField
[ ] 0023_alter_requirements_file_field
[ ] 0024_remove_collectionversion_certification
[ ] 0025_increase_collection_version_version_size
[ ] 0026_deprecation_per_repository
[ ] 0027_tag_length
[ ] 0028_collectionversion_namespace_length
[ ] 0029_manifest_and_files_json_fields
[ ] 0030_collectionversion_requires_ansible
[ ] 0031_ansiblerepository_last_synced_metadata_time
[ ] 0032_collectionremote_sync_dependencies
[ ] 0033_swap_distribution_model
[ ] 0034_handle_jsonfield_warnings
[ ] 0035_deprecation_content
[ ] 0036_update_repository_content
[ ] 0037_gitremote
[ ] 0038_collectionversionsignature
[ ] 0039_collectionremote_signed_only
[ ] 0040_ansiblerepository_keyring
[ ] 0041_alter_collectionversion_collection
[ ] 0042_ansiblerepository_gpgkey
[ ] 0043_alter_collectionversionsignature_data
[ ] 0044_alter_collectionremote_token
[ ] 0045_downloadlog
[ ] 0046_add_fulltext_search_fix
[ ] 0047_ansible_namespace
[ ] 0048_collectionversionmark
[ ] 0049_rbac_permissions
[ ] 0050_crossrepositorycollectionversionindex
[ ] 0051_cvindex_build
[ ] 0052_alter_ansiblecollectiondeprecated_content_ptr_and_more
[ ] 0053_collectiondownloadcount
[ ] 0054_split_collection_version_numbers
[ ] 0055_alter_collectionversion_version_alter_role_version
[ ] 0001_initial
[ ] 0002_alter_permission_name_max_length
[ ] 0003_alter_user_email_max_length
[ ] 0004_alter_user_username_opts
[ ] 0005_alter_user_last_login_null
[ ] 0006_require_contenttypes_0002
[ ] 0007_alter_validators_add_error_messages
[ ] 0008_alter_user_username_max_length
[ ] 0009_alter_user_last_name_max_length
[ ] 0010_alter_group_name_max_length
[ ] 0011_update_proxy_permissions
[ ] 0012_alter_user_first_name_max_length
[ ] 0001_initial
[ ] 0002_alter_rhsmcertguard_contentguard_ptr_and_more
[ ] 0001_initial
[ ] 0002_containerrepository
[ ] 0003_oci_mediatype
[ ] 0004_upload
[ ] 0005_contentredirectcontentguard
[ ] 0006_containerpushrepository
[ ] 0007_clear_tags_artifacts_refs
[ ] 0008_include_exclude_tags
[ ] 0009_container_namespace
[ ] 0010_remove_uploadchunk
[ ] 0011_add_container_repository_permissions
[ ] 0012_add_container_namespace_permissions
[ ] 0013_add_pull_push_permissions
[ ] 0014_containerdistribution_private
[ ] 0015_manage_tags_push_repo
[ ] 0016_add_delete_versions_permission
[ ] 0017_add_granular_perms
[ ] 0018_containerdistribution_description
[ ] 0019_DATA_distribution_model_swap
[ ] 0020_update_push_repo_perms
[ ] 0021_data_move_redirect_content_guard_to_core
[ ] 0022_delete_contentredirectcontentguard
[ ] 0023_manifestsignature
[ ] 0024_containerremote_sigstore
[ ] 0025_signature_stored_in_textfield
[ ] 0026_manifest_signing_service
[ ] 0027_data_translate_perms_to_roles
[ ] 0028_add_role_manage_permissions
[ ] 0029_remove_blob_media_type
[ ] 0030_enforce_tagged_manifest_reference
[ ] 0031_replace_charf_with_textf
[ ] 0032_upload_artifact
[ ] 0033_raise_warning_for_repair
[ ] 0034_translate_signed_schema
[ ] 0035_alter_blob_content_ptr_and_more
[ ] 0036_containerpushrepository_pending_blobs_manifests
[ ] 0037_create_pull_through_cache_models
[ ] 0038_add_manifest_metadata_fields
[ ] 0039_manifest_data
[ ] 0001_initial
[ ] 0002_remove_content_type_name
[ ] 0001_initial
[ ] 0002_increase_artifact_size_field
[ ] 0003_remove_upload_completed
[ ] 0004_add_duplicated_reserved_resources
[ ] 0005_progressreport_code
[ ] 0006_repository_plugin_managed
[ ] 0007_delete_progress_proxies
[ ] 0008_published_metadata_as_content
[ ] 0009_remove_task_non_fatal_errors
[ ] 0010_pulp_fields
[ ] 0011_relative_path
[ ] 0012_auto_20191104_2000
[ ] 0013_repository_pulp_type
[ ] 0014_remove_repository_plugin_managed
[ ] 0015_auto_20191112_1426
[ ] 0016_charfield_to_textfield
[ ] 0017_remove_task_parent
[ ] 0018_auto_20191127_2350
[ ] 0019_add_signing_service_model
[ ] 0020_change_publishedartifact_constraints
[ ] 0021_add_signing_service_foreign_key
[ ] 0022_rename_last_version
[ ] 0023_change_exporter_models
[ ] 0024_use_local_storage_for_uploads
[ ] 0025_task_parent_task
[ ] 0026_task_group
[ ] 0027_export_backend
[ ] 0028_import_importer_pulpimporter_pulpimporterrepository
[ ] 0029_export_delete
[ ] 0030_taskgroup_all_tasks_dispatched
[ ] 0031_import_export_validate_params
[ ] 0032_export_to_chunks
[ ] 0033_increase_remote_artifact_size_field
[ ] 0034_groupprogressreport
[ ] 0035_content_upstream_id
[ ] 0036_unprotect_last_export
[ ] 0037_pulptemporaryfile
[ ] 0038_repository_remote
[ ] 0039_change_download_concurrency
[ ] 0040_set_admin_is_staff
[ ] 0041_accesspolicy
[ ] 0042_rbac_for_tasks
[ ] 0043_toc_attribute
[ ] 0044_temp_file_artifact_field
[ ] 0045_accesspolicy_permissions_allow_null
[ ] 0046_task__resource_job_id
[ ] 0047_improve_orphan_cleanup
[ ] 0048_fips_checksums
[ ] 0049_add_file_field_to_uploadchunk
[ ] 0050_namespace_access_policies
[ ] 0051_timeoutfields
[ ] 0052_tasking_logging_cid
[ ] 0053_remote_headers
[ ] 0054_add_public_key
[ ] 0055_label
[ ] 0056_remote_rate_limit
[ ] 0057_add_label_indexes
[ ] 0058_accesspolicy_customized
[ ] 0059_proxy_creds
[ ] 0060_data_migration_proxy_creds
[ ] 0061_call_handle_artifact_checksums_command
[ ] 0062_add_new_distribution_mastermodel
[ ] 0063_repository_retained_versions
[ ] 0064_repository_user_hidden
[ ] 0064_add_new_style_task_columns
[ ] 0065_merge_20210615_1211
[ ] 0066_download_concurrency_and_retry_changes
[ ] 0067_add_protect_to_task_reservation
[ ] 0068_add_timestamp_of_interest
[ ] 0069_update_json_fields
[ ] 0070_rename_retained_versions
[ ] 0071_filesystemexport_filesystemexporter
[ ] 0072_add_method_to_filesystem_exporter
[ ] 0073_encrypt_remote_fields
[ ] 0074_acs
[ ] 0075_rbaccontentguard
[ ] 0076_remove_reserved_resource
[ ] 0077_move_remote_url_credentials
[ ] 0078_grouprole_role_userrole
[ ] 0079_rename_permissions_assignment_accesspolicy_creation_hooks
[ ] 0080_proxy_group_model
[ ] 0081_reapplabel_group_permissions
[ ] 0082_add_manage_roles_permissions
[ ] 0083_alter_group_options
[ ] 0084_alter_rbaccontentguard_options
[ ] 0085_contentredirectcontentguard
[ ] 0086_task_json_fields
[ ] 0087_taskschedule
[ ] 0088_accesspolicy_queryset_scoping
[ ] 0089_alter_contentredirectcontentguard_options
[ ] 0090_char_to_text_field
[ ] 0091_systemid
[ ] 0092_alter_upload_options
[ ] 0093_add_info_field_repositoryversion
[ ] 0094_protect_repository_content
[ ] 0095_artifactdistribution
[ ] 0096_alter_task_logging_cid
[ ] 0097_remove_telemetry_task_schedule
[ ] 0098_pulp_labels
[ ] 0099_versions_field
[ ] 0100_upstreampulp
[ ] 0101_add_domain
[ ] 0102_add_domain_relations
[ ] 0103_alter_export_task
[ ] 0104_delete_label
[ ] 0105_abstract_uuid_gen
[ ] 0106_alter_artifactdistribution_distribution_ptr_and_more
[ ] 0107_distribution_hidden
[ ] 0108_task_versions
[ ] 0109_contentartifact_relative_path_index
[ ] 0110_apiappstatus
[ ] 0111_task_enc_args_task_enc_kwargs
[ ] 0112_alter_upstreampulp_options
[ ] 0113_headercontentguard
[ ] 0114_remove_task_args_remove_task_kwargs
[ ] 0115_compositecontentguard
[ ] 0116_alter_remoteartifact_md5_alter_remoteartifact_sha1_and_more
[ ] 0117_task_unblocked_at
[ ] 0118_task_core_task_unblock_2276a4_idx_and_more
[ ] 0001_initial
[ ] 0002_auto_20190905_1000
[ ] 0003_rename_package_field
[ ] 0004_auto_20191014_1722
[ ] 0005_rename_release_file
[ ] 0006_debrepository
[ ] 0007_create_metadata_models
[ ] 0008_debremote_gpgkey
[ ] 0009_apt_release_signing_service
[ ] 0010_debpublication_signing_service
[ ] 0011_rename_models_to_apt
[ ] 0012_auto_20200803_1337
[ ] 0013_aptremote_ignore_missing_package_indices
[ ] 0014_swap_distribution_model
[ ] 0015_add_custom_fields_to_packages
[ ] 0016_switch_to_newer_jsonfield_model
[ ] 0017_allow_longer_string_lists
[ ] 0018_textfield_conversion
[ ] 0019_immutable_metadata_constraints
[ ] 0020_remove_fk_relations_in_packageindex_and_installerfileindex
[ ] 0021_remove_release_from_structure_types
[ ] 0022_alter_aptdistribution_distribution_ptr_and_more
[ ] 0023_add_default_signing_services
[ ] 0024_add_release_fields
[ ] 0025_merge_colliding_structure_content
[ ] 0026_aptrepository_publish_upstream_release_fields
[ ] 0027_alter_aptpublication_structured
[ ] 0028_sourcepackage_sourcepackagereleasecomponent_and_more
[ ] 0029_distributedpublication
[ ] 0001_initial
[ ] 0002_file_related_names
[ ] 0003_auto_20191014_1721
[ ] 0004_filefilesystemexporter
[ ] 0005_filerepository
[ ] 0006_delete_filefilesystemexporter
[ ] 0007_filefilesystemexporter
[ ] 0008_add_manifest_field
[ ] 0009_move_data_to_new_master_distribution_model
[ ] 0010_auto_publish
[ ] 0011_fix_auto_publish
[ ] 0012_delete_filefilesystemexporter
[ ] 0013_file_acs
[ ] 0014_new_rbac_permissions
[ ] 0015_allow_null_manifest
[ ] 0016_add_domain
[ ] 0017_alter_filealternatecontentsource_alternatecontentsource_ptr_and_more
[ ] 0001_initial
[ ] 0002_maven_related_names
[ ] 0003_mavenrepository
[ ] 0004_swap_distribution_models
[ ] 0005_mavenmetadata
[ ] 0006_alter_mavenartifact_content_ptr_and_more
[ ] 0001_initial
[ ] 0002_add_relative_path_uniqueness
[ ] 0003_create_many_to_many_objs_commits
[ ] 0004_add_include_exclude_refs
[ ] 0005_add_static_delta_support
[ ] 0006_alter_pointers_to_related_models_globally
[ ] 0007_add_model_permissions
[ ] 0008_add_domain_support
[ ] 0001_initial
[ ] 0002_pythonpackagecontent_python_version
[ ] 0003_new_sync_filters
[ ] 0004_DATA_swap_distribution_model
[ ] 0005_pythonpackagecontent_sha256
[ ] 0006_pythonrepository_autopublish
[ ] 0007_pythonpackagecontent_mv-2-1
[ ] 0008_pythonpackagecontent_unique_sha256
[ ] 0009_pythondistribution_allow_uploads
[ ] 0010_update_json_field
[ ] 0011_alter_pythondistribution_distribution_ptr_and_more
[ ] 0012_add_domain
[ ] 0013_add_rbac_permissions
[ ] 0001_initial
[ ] 0002_updaterecord_reboot_suggested
[ ] 0003_DATA_incorrect_json
[ ] 0004_add_metadata_signing_service_fk
[ ] 0005_optimize_sync
[ ] 0006_opensuse_support
[ ] 0007_checksum_types
[ ] 0008_advisory_pkg_sumtype_as_int
[ ] 0009_revision_null
[ ] 0010_revision_null_redo
[ ] 0011_rpmremote_sles_auth_token
[ ] 0012_remove_pkg_group_env_cat_related_pkgs
[ ] 0013_RAW_rpm_evr_extension
[ ] 0014_rpmrepository_package_retention_policy
[ ] 0015_repo_metadata
[ ] 0016_dist_tree_nofk
[ ] 0017_merge_advisory_collections
[ ] 0018_updatecollection__update_record
[ ] 0019_migrate_updatecollection_data
[ ] 0020_remove_updatecollection_m2m
[ ] 0021_rename_updatecollection_update_record
[ ] 0022_add_collections_related_name
[ ] 0023_increase_distribution_release_short
[ ] 0024_change_subrepo_relation_properties
[ ] 0025_remove_orphaned_subrepos
[ ] 0026_add_gpgcheck_options
[ ] 0027_checksum_null
[ ] 0028_rpmrepository_last_sync_repomd_cheksum
[ ] 0029_rpmpublication_sqlite_metadata
[ ] 0030_DATA_fix_updaterecord
[ ] 0031_modulemd_static_context
[ ] 0032_ulnremote
[ ] 0033_new_distribution_model
[ ] 0034_auto_publish
[ ] 0035_fix_auto_publish
[ ] 0036_checksum_type
[ ] 0037_DATA_remove_rpmrepository_sub_repo
[ ] 0037_update_json_field
[ ] 0038_fix_sync_optimization
[ ] 0039_disttree_digest
[ ] 0040_rpmalternatecontentsource
[ ] 0041_modulemdobsolete
[ ] 0042_alter_repometadatafile_data_type
[ ] 0043_textfield_conversion
[ ] 0044_noartifact_modules
[ ] 0045_modulemd_fields
[ ] 0046_rbac_perms
[ ] 0047_modulemd_datefield
[ ] 0048_artifacts_dependencies_fix
[ ] 0049_profiles_fix
[ ] 0050_alter_addon_pulp_id_alter_checksum_pulp_id_and_more
[ ] 0051_alter_distributiontree_unique_together_and_more
[ ] 0052_modulemd_digest
[ ] 0053_rpmdistribution_generate_repo_config
[ ] 0054_remove_gpg_fields
[ ] 0055_add_repo_config_field
[ ] 0056_remove_rpmpublication_sqlite_metadata_and_more
[ ] 0057_rpmpublication_checksum_type_and_more
[ ] 0058_alter_addon_repository_alter_variant_repository
[ ] 0059_rpmpublication_compression_type_and_more
[ ] 0060_rpmpublication_compression_type_empty
[ ] 0061_fix_modulemd_defaults_digest
[ ] 0062_rpmpackagesigningservice_and_more
[ ] 0001_initial
sh-5.1$

##############################33

sh-5.1$ /usr/local/bin/pulpcore-manager showmigrations | grep ‘[ ]’

[ ] 0001_initial
[ ] 0002_advanced_collections
[ ] 0003_add_tags_and_collectionversion_fields
[ ] 0004_add_fulltext_search_indexes
[ ] 0005_collectionversion_is_highest
[ ] 0006_remove_whitelist_and_alter_collection_version_name
[ ] 0007_collectionversion_is_certified
[ ] 0008_collectionremote_requirements_file
[ ] 0009_collectionimport
[ ] 0010_ansible_related_names
[ ] 0011_collectionimport
[ ] 0012_auto_20190906_2253
[ ] 0013_pulp_fields
[ ] 0014_certification_enum
[ ] 0015_ansiblerepository
[ ] 0016_add_extension
[ ] 0017_increase_length_collectionversion_fields
[ ] 0018_fix_collection_relative_path
[ ] 0019_collection_token
[ ] 0020_auto_20200810_1926
[ ] 0021_rename_role_remote
[ ] 0022_URLField_to_CharField
[ ] 0023_alter_requirements_file_field
[ ] 0024_remove_collectionversion_certification
[ ] 0025_increase_collection_version_version_size
[ ] 0026_deprecation_per_repository
[ ] 0027_tag_length
[ ] 0028_collectionversion_namespace_length
[ ] 0029_manifest_and_files_json_fields
[ ] 0030_collectionversion_requires_ansible
[ ] 0031_ansiblerepository_last_synced_metadata_time
[ ] 0032_collectionremote_sync_dependencies
[ ] 0033_swap_distribution_model
[ ] 0034_handle_jsonfield_warnings
[ ] 0035_deprecation_content
[ ] 0036_update_repository_content
[ ] 0037_gitremote
[ ] 0038_collectionversionsignature
[ ] 0039_collectionremote_signed_only
[ ] 0040_ansiblerepository_keyring
[ ] 0041_alter_collectionversion_collection
[ ] 0042_ansiblerepository_gpgkey
[ ] 0043_alter_collectionversionsignature_data
[ ] 0044_alter_collectionremote_token
[ ] 0045_downloadlog
[ ] 0046_add_fulltext_search_fix
[ ] 0047_ansible_namespace
[ ] 0048_collectionversionmark
[ ] 0049_rbac_permissions
[ ] 0050_crossrepositorycollectionversionindex
[ ] 0051_cvindex_build
[ ] 0052_alter_ansiblecollectiondeprecated_content_ptr_and_more
[ ] 0053_collectiondownloadcount
[ ] 0054_split_collection_version_numbers
[ ] 0055_alter_collectionversion_version_alter_role_version
[ ] 0001_initial
[ ] 0002_alter_permission_name_max_length
[ ] 0003_alter_user_email_max_length
[ ] 0004_alter_user_username_opts
[ ] 0005_alter_user_last_login_null
[ ] 0006_require_contenttypes_0002
[ ] 0007_alter_validators_add_error_messages
[ ] 0008_alter_user_username_max_length
[ ] 0009_alter_user_last_name_max_length
[ ] 0010_alter_group_name_max_length
[ ] 0011_update_proxy_permissions
[ ] 0012_alter_user_first_name_max_length
[ ] 0001_initial
[ ] 0002_alter_rhsmcertguard_contentguard_ptr_and_more
[ ] 0001_initial
[ ] 0002_containerrepository
[ ] 0003_oci_mediatype
[ ] 0004_upload
[ ] 0005_contentredirectcontentguard
[ ] 0006_containerpushrepository
[ ] 0007_clear_tags_artifacts_refs
[ ] 0008_include_exclude_tags
[ ] 0009_container_namespace
[ ] 0010_remove_uploadchunk
[ ] 0011_add_container_repository_permissions
[ ] 0012_add_container_namespace_permissions
[ ] 0013_add_pull_push_permissions
[ ] 0014_containerdistribution_private
[ ] 0015_manage_tags_push_repo
[ ] 0016_add_delete_versions_permission
[ ] 0017_add_granular_perms
[ ] 0018_containerdistribution_description
[ ] 0019_DATA_distribution_model_swap
[ ] 0020_update_push_repo_perms
[ ] 0021_data_move_redirect_content_guard_to_core
[ ] 0022_delete_contentredirectcontentguard
[ ] 0023_manifestsignature
[ ] 0024_containerremote_sigstore
[ ] 0025_signature_stored_in_textfield
[ ] 0026_manifest_signing_service
[ ] 0027_data_translate_perms_to_roles
[ ] 0028_add_role_manage_permissions
[ ] 0029_remove_blob_media_type
[ ] 0030_enforce_tagged_manifest_reference
[ ] 0031_replace_charf_with_textf
[ ] 0032_upload_artifact
[ ] 0033_raise_warning_for_repair
[ ] 0034_translate_signed_schema
[ ] 0035_alter_blob_content_ptr_and_more
[ ] 0036_containerpushrepository_pending_blobs_manifests
[ ] 0037_create_pull_through_cache_models
[ ] 0038_add_manifest_metadata_fields
[ ] 0039_manifest_data
[ ] 0001_initial
[ ] 0002_remove_content_type_name
[ ] 0001_initial
[ ] 0002_increase_artifact_size_field
[ ] 0003_remove_upload_completed
[ ] 0004_add_duplicated_reserved_resources
[ ] 0005_progressreport_code
[ ] 0006_repository_plugin_managed
[ ] 0007_delete_progress_proxies
[ ] 0008_published_metadata_as_content
[ ] 0009_remove_task_non_fatal_errors
[ ] 0010_pulp_fields
[ ] 0011_relative_path
[ ] 0012_auto_20191104_2000
[ ] 0013_repository_pulp_type
[ ] 0014_remove_repository_plugin_managed
[ ] 0015_auto_20191112_1426
[ ] 0016_charfield_to_textfield
[ ] 0017_remove_task_parent
[ ] 0018_auto_20191127_2350
[ ] 0019_add_signing_service_model
[ ] 0020_change_publishedartifact_constraints
[ ] 0021_add_signing_service_foreign_key
[ ] 0022_rename_last_version
[ ] 0023_change_exporter_models
[ ] 0024_use_local_storage_for_uploads
[ ] 0025_task_parent_task
[ ] 0026_task_group
[ ] 0027_export_backend
[ ] 0028_import_importer_pulpimporter_pulpimporterrepository
[ ] 0029_export_delete
[ ] 0030_taskgroup_all_tasks_dispatched
[ ] 0031_import_export_validate_params
[ ] 0032_export_to_chunks
[ ] 0033_increase_remote_artifact_size_field
[ ] 0034_groupprogressreport
[ ] 0035_content_upstream_id
[ ] 0036_unprotect_last_export
[ ] 0037_pulptemporaryfile
[ ] 0038_repository_remote
[ ] 0039_change_download_concurrency
[ ] 0040_set_admin_is_staff
[ ] 0041_accesspolicy
[ ] 0042_rbac_for_tasks
[ ] 0043_toc_attribute
[ ] 0044_temp_file_artifact_field
[ ] 0045_accesspolicy_permissions_allow_null
[ ] 0046_task__resource_job_id
[ ] 0047_improve_orphan_cleanup
[ ] 0048_fips_checksums
[ ] 0049_add_file_field_to_uploadchunk
[ ] 0050_namespace_access_policies
[ ] 0051_timeoutfields
[ ] 0052_tasking_logging_cid
[ ] 0053_remote_headers
[ ] 0054_add_public_key
[ ] 0055_label
[ ] 0056_remote_rate_limit
[ ] 0057_add_label_indexes
[ ] 0058_accesspolicy_customized
[ ] 0059_proxy_creds
[ ] 0060_data_migration_proxy_creds
[ ] 0061_call_handle_artifact_checksums_command
[ ] 0062_add_new_distribution_mastermodel
[ ] 0063_repository_retained_versions
[ ] 0064_repository_user_hidden
[ ] 0064_add_new_style_task_columns
[ ] 0065_merge_20210615_1211
[ ] 0066_download_concurrency_and_retry_changes
[ ] 0067_add_protect_to_task_reservation
[ ] 0068_add_timestamp_of_interest
[ ] 0069_update_json_fields
[ ] 0070_rename_retained_versions
[ ] 0071_filesystemexport_filesystemexporter
[ ] 0072_add_method_to_filesystem_exporter
[ ] 0073_encrypt_remote_fields
[ ] 0074_acs
[ ] 0075_rbaccontentguard
[ ] 0076_remove_reserved_resource
[ ] 0077_move_remote_url_credentials
[ ] 0078_grouprole_role_userrole
[ ] 0079_rename_permissions_assignment_accesspolicy_creation_hooks
[ ] 0080_proxy_group_model
[ ] 0081_reapplabel_group_permissions
[ ] 0082_add_manage_roles_permissions
[ ] 0083_alter_group_options
[ ] 0084_alter_rbaccontentguard_options
[ ] 0085_contentredirectcontentguard
[ ] 0086_task_json_fields
[ ] 0087_taskschedule
[ ] 0088_accesspolicy_queryset_scoping
[ ] 0089_alter_contentredirectcontentguard_options
[ ] 0090_char_to_text_field
[ ] 0091_systemid
[ ] 0092_alter_upload_options
[ ] 0093_add_info_field_repositoryversion
[ ] 0094_protect_repository_content
[ ] 0095_artifactdistribution
[ ] 0096_alter_task_logging_cid
[ ] 0097_remove_telemetry_task_schedule
[ ] 0098_pulp_labels
[ ] 0099_versions_field
[ ] 0100_upstreampulp
[ ] 0101_add_domain
[ ] 0102_add_domain_relations
[ ] 0103_alter_export_task
[ ] 0104_delete_label
[ ] 0105_abstract_uuid_gen
[ ] 0106_alter_artifactdistribution_distribution_ptr_and_more
[ ] 0107_distribution_hidden
[ ] 0108_task_versions
[ ] 0109_contentartifact_relative_path_index
[ ] 0110_apiappstatus
[ ] 0111_task_enc_args_task_enc_kwargs
[ ] 0112_alter_upstreampulp_options
[ ] 0113_headercontentguard
[ ] 0114_remove_task_args_remove_task_kwargs
[ ] 0115_compositecontentguard
[ ] 0116_alter_remoteartifact_md5_alter_remoteartifact_sha1_and_more
[ ] 0117_task_unblocked_at
[ ] 0118_task_core_task_unblock_2276a4_idx_and_more
[ ] 0001_initial
[ ] 0002_auto_20190905_1000
[ ] 0003_rename_package_field
[ ] 0004_auto_20191014_1722
[ ] 0005_rename_release_file
[ ] 0006_debrepository
[ ] 0007_create_metadata_models
[ ] 0008_debremote_gpgkey
[ ] 0009_apt_release_signing_service
[ ] 0010_debpublication_signing_service
[ ] 0011_rename_models_to_apt
[ ] 0012_auto_20200803_1337
[ ] 0013_aptremote_ignore_missing_package_indices
[ ] 0014_swap_distribution_model
[ ] 0015_add_custom_fields_to_packages
[ ] 0016_switch_to_newer_jsonfield_model
[ ] 0017_allow_longer_string_lists
[ ] 0018_textfield_conversion
[ ] 0019_immutable_metadata_constraints
[ ] 0020_remove_fk_relations_in_packageindex_and_installerfileindex
[ ] 0021_remove_release_from_structure_types
[ ] 0022_alter_aptdistribution_distribution_ptr_and_more
[ ] 0023_add_default_signing_services
[ ] 0024_add_release_fields
[ ] 0025_merge_colliding_structure_content
[ ] 0026_aptrepository_publish_upstream_release_fields
[ ] 0027_alter_aptpublication_structured
[ ] 0028_sourcepackage_sourcepackagereleasecomponent_and_more
[ ] 0029_distributedpublication
[ ] 0001_initial
[ ] 0002_file_related_names
[ ] 0003_auto_20191014_1721
[ ] 0004_filefilesystemexporter
[ ] 0005_filerepository
[ ] 0006_delete_filefilesystemexporter
[ ] 0007_filefilesystemexporter
[ ] 0008_add_manifest_field
[ ] 0009_move_data_to_new_master_distribution_model
[ ] 0010_auto_publish
[ ] 0011_fix_auto_publish
[ ] 0012_delete_filefilesystemexporter
[ ] 0013_file_acs
[ ] 0014_new_rbac_permissions
[ ] 0015_allow_null_manifest
[ ] 0016_add_domain
[ ] 0017_alter_filealternatecontentsource_alternatecontentsource_ptr_and_more
[ ] 0001_initial
[ ] 0002_maven_related_names
[ ] 0003_mavenrepository
[ ] 0004_swap_distribution_models
[ ] 0005_mavenmetadata
[ ] 0006_alter_mavenartifact_content_ptr_and_more
[ ] 0001_initial
[ ] 0002_add_relative_path_uniqueness
[ ] 0003_create_many_to_many_objs_commits
[ ] 0004_add_include_exclude_refs
[ ] 0005_add_static_delta_support
[ ] 0006_alter_pointers_to_related_models_globally
[ ] 0007_add_model_permissions
[ ] 0008_add_domain_support
[ ] 0001_initial
[ ] 0002_pythonpackagecontent_python_version
[ ] 0003_new_sync_filters
[ ] 0004_DATA_swap_distribution_model
[ ] 0005_pythonpackagecontent_sha256
[ ] 0006_pythonrepository_autopublish
[ ] 0007_pythonpackagecontent_mv-2-1
[ ] 0008_pythonpackagecontent_unique_sha256
[ ] 0009_pythondistribution_allow_uploads
[ ] 0010_update_json_field
[ ] 0011_alter_pythondistribution_distribution_ptr_and_more
[ ] 0012_add_domain
[ ] 0013_add_rbac_permissions
[ ] 0001_initial
[ ] 0002_updaterecord_reboot_suggested
[ ] 0003_DATA_incorrect_json
[ ] 0004_add_metadata_signing_service_fk
[ ] 0005_optimize_sync
[ ] 0006_opensuse_support
[ ] 0007_checksum_types
[ ] 0008_advisory_pkg_sumtype_as_int
[ ] 0009_revision_null
[ ] 0010_revision_null_redo
[ ] 0011_rpmremote_sles_auth_token
[ ] 0012_remove_pkg_group_env_cat_related_pkgs
[ ] 0013_RAW_rpm_evr_extension
[ ] 0014_rpmrepository_package_retention_policy
[ ] 0015_repo_metadata
[ ] 0016_dist_tree_nofk
[ ] 0017_merge_advisory_collections
[ ] 0018_updatecollection__update_record
[ ] 0019_migrate_updatecollection_data
[ ] 0020_remove_updatecollection_m2m
[ ] 0021_rename_updatecollection_update_record
[ ] 0022_add_collections_related_name
[ ] 0023_increase_distribution_release_short
[ ] 0024_change_subrepo_relation_properties
[ ] 0025_remove_orphaned_subrepos
[ ] 0026_add_gpgcheck_options
[ ] 0027_checksum_null
[ ] 0028_rpmrepository_last_sync_repomd_cheksum
[ ] 0029_rpmpublication_sqlite_metadata
[ ] 0030_DATA_fix_updaterecord
[ ] 0031_modulemd_static_context
[ ] 0032_ulnremote
[ ] 0033_new_distribution_model
[ ] 0034_auto_publish
[ ] 0035_fix_auto_publish
[ ] 0036_checksum_type
[ ] 0037_DATA_remove_rpmrepository_sub_repo
[ ] 0037_update_json_field
[ ] 0038_fix_sync_optimization
[ ] 0039_disttree_digest
[ ] 0040_rpmalternatecontentsource
[ ] 0041_modulemdobsolete
[ ] 0042_alter_repometadatafile_data_type
[ ] 0043_textfield_conversion
[ ] 0044_noartifact_modules
[ ] 0045_modulemd_fields
[ ] 0046_rbac_perms
[ ] 0047_modulemd_datefield
[ ] 0048_artifacts_dependencies_fix
[ ] 0049_profiles_fix
[ ] 0050_alter_addon_pulp_id_alter_checksum_pulp_id_and_more
[ ] 0051_alter_distributiontree_unique_together_and_more
[ ] 0052_modulemd_digest
[ ] 0053_rpmdistribution_generate_repo_config
[ ] 0054_remove_gpg_fields
[ ] 0055_add_repo_config_field
[ ] 0056_remove_rpmpublication_sqlite_metadata_and_more
[ ] 0057_rpmpublication_checksum_type_and_more
[ ] 0058_alter_addon_repository_alter_variant_repository
[ ] 0059_rpmpublication_compression_type_and_more
[ ] 0060_rpmpublication_compression_type_empty
[ ] 0061_fix_modulemd_defaults_digest
[ ] 0062_rpmpackagesigningservice_and_more
[ ] 0001_initial

Both are from 2 content pods

#11

Considering that both outputs are the same, I’m discarding the possibility of the pods connecting to different dbs.
It seems like no migrations ran at all.
Can you please check if the following command will return any error

oc exec pulp-worker-74fcf5dcb9-bdthr -- /usr/local/bin/pulpcore-manager migrate
#12

I have executed that in one my cluster . Pods are up , however all repositories (remote and local) are empty after that .This specific cluster has synced multiple redhat repos at the moment and would like to keep the data.

#13

Considering the number of pending migrations, it seems like when you deleted the db pod all your data was also lost.

Can you provide us the output of the following commands in https://pastebin.centos.org?

oc get pod pulp-server-nonprod-database-0 -oyaml
oc exec pulp-server-nonprod-database-0 -- mount|grep postgres
oc exec pulp-server-nonprod-database-0 -- sh -c 'echo $PGDATA'
oc exec pulp-server-nonprod-database-0 -- sh -c 'ls $PGDATA'
#14

Updated the details - pulp-openshift-operator-worker-nodes-stuck-in-db-m - Pastebin Service

#15

Disabling init container running database migration wont help as pods show below erro.

pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘8@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘8@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘7@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘7@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘8@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘8@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘7@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘7@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘8@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘8@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘7@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘7@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘8@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘8@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘7@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.
pulp [None]: pulpcore.app.entrypoint:INFO: Api App ‘7@pulp-worker-8668944fc7-65b9x’ failed to write a heartbeat to the database, sleeping for ‘45.0’ seconds.

#16

hum… there is no PVC attached for the postgres data (it is using the emptyDir volume):

...
    volumeMounts:
    - mountPath: /var/lib/postgresql/data
      name: pulp-server-nonprod-postgres
...
  volumes:
  - emptyDir: {}
    name: pulp-server-nonprod-postgres

that is why when you deleted the pod all of its data is lost and we had to run the migrations again.

Can you share your Pulp CR (in pastebin, please), so we can see how the operator is configured to deploy the db?

oc get pulp -oyaml

Here is a doc on how to configure pulp operator to deploy a postgres instance with PVC: Pulp Operator storage configuration - Pulp Project

#17

Updated in pastebin - Pulp operator database issue - Pastebin Service

#18

ok, I can see that you provided a StorageClass for your pulpcore pods:

    file_storage_access_mode: ReadWriteMany
    file_storage_size: 50Gi
    file_storage_storage_class: ocs-storagecluster-cephfs

but there is no definition of storage for the database pods:

    database:
      postgres_resource_requirements:
        limits:
          cpu: 4
          memory: 4Gi
        requests:
          cpu: 250m
          memory: 256Mi

to use a StorageClass for the db pods you need to provide the database.postgres_storage_class field, for example:

    database:
      postgres_storage_class: <name of the storage class>
      postgres_resource_requirements:
        limits:
          cpu: 4
          memory: 4Gi
        requests:
          cpu: 250m
          memory: 256Mi
#19

Thank you for the findings.
postgres_storage_class was removed as it throwed below error

2024-07-05T16:57:43Z INFO repo_manager/database.go:104 The pulp-server-nonprod-database StatefulSet has been modified! Reconciling …
2024-07-05T16:57:43Z ERROR repo_manager/database.go:113 Error trying to update the pulp-server-nonprod-database StatefulSet object … {“error”: “StatefulSet.apps “pulp-server-nonprod-database” is invalid: spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘ordinals’, ‘template’, ‘updateStrategy’, ‘persistentVolumeClaimRetentionPolicy’ and ‘minReadySeconds’ are forbidden”}

However setting pvc for databae (removing postgres_storage_class) looks fine at the moment.

current config
database:
postgres_resource_requirements:
limits:
cpu: 4
memory: 4Gi
requests:
cpu: 250m
memory: 256Mi
pvc: pulp-server-nonprod-file-storage

Any suggestions ?

#20

This error means that the operator tried to update the volumeClaimTemplates field from StatefulSet, but this field is immutable.
To workaround this error, you can try to configure your Pulp CR with the database.postgres_storage_class again and delete the pulp-server-nonprod-database StatefulSet (your running database pod will be removed). The operator should now recreate a new StatefulSet with the expected StorageClass.