Pulp upgrading from version 3.88 to 3.89 and above causes migration error

Support
#1

Problem:
Pulp upgrading from version 3.88 to 3.89 and above causes migration error

Expected outcome:
Successful migration from 3.88 to the latest version

Pulpcore version:
“core”: “3.88.0”

Pulp plugins installed and their versions:
“deb”: “3.7.0”,
“npm”: “0.4.0”,
“rpm”: “3.32.0”,
“core”: “3.88.0”,
“file”: “3.88.0”,
“maven”: “0.11.0”,
“ostree”: “2.5.0”,
“python”: “3.19.0”,
“ansible”: “0.28.0”,
“certguard”: “3.88.0”,
“container”: “2.26.1”

Operating system - distribution and version:
Photon 5 - Docker Compose - Multi-Container

Other relevant data:
When upgrading from Pulp core version 3.88 to 3.89 and above (including 3.90) a migration error is reported:

Database migration in progress. Waiting…
migration_service-1 | Operations to perform:
migration_service-1 | Apply all migrations: ansible, auth, certguard, container, contenttypes, core, deb, file, maven, npm, ostree, python, rpm, sessions
migration_service-1 | Running migrations:
migration_service-1 | Traceback (most recent call last):
migration_service-1 | File “/usr/local/bin/pulpcore-manager”, line 8, in
migration_service-1 | sys.exit(manage())
migration_service-1 | ^^^^^^^^
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/pulpcore/app/manage.py”, line 11, in manage
migration_service-1 | execute_from_command_line(sys.argv)
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/core/management/init.py”, line 442, in execute_from_command_line
migration_service-1 | utility.execute()
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/core/management/init.py”, line 436, in execute
migration_service-1 | self.fetch_command(subcommand).run_from_argv(self.argv)
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/core/management/base.py”, line 412, in run_from_argv
migration_service-1 | self.execute(*args, **cmd_options)
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/core/management/base.py”, line 458, in execute
migration_service-1 | output = self.handle(*args, **options)
migration_service-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/core/management/base.py”, line 106, in wrapper
migration_service-1 | res = handle_func(*args, **kwargs)
migration_service-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/core/management/commands/migrate.py”, line 356, in handle
migration_service-1 | post_migrate_state = executor.migrate(
migration_service-1 | ^^^^^^^^^^^^^^^^^
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py”, line 135, in migrate
migration_service-1 | state = self._migrate_all_forwards(
migration_service-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py”, line 167, in _migrate_all_forwards
migration_service-1 | state = self.apply_migration(
migration_service-1 | ^^^^^^^^^^^^^^^^^^^^^
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py”, line 252, in apply_migration
migration_service-1 | state = migration.apply(state, schema_editor)
migration_service-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py”, line 132, in apply
migration_service-1 | operation.database_forwards(
migration_service-1 | File “/usr/local/lib/python3.11/site-packages/pulpcore/migrations.py”, line 74,

RuntimeError: Incompatible versions detected (core >= 3.87 needed):
migration_service-1 | - ‘core’=‘3.86.0’ with pulp worker ‘20@pulp-api’
migration_service-1 | - ‘core’=‘3.86.0’ with pulp worker ‘15@pulp-api’
migration_service-1 | - ‘core’=‘3.86.0’ with pulp worker ‘21@pulp-api’
migration_service-1 | - ‘core’=‘3.86.0’ with pulp worker ‘16@pulp-api’
migration_service-1 | - ‘core’=‘3.86.0’ with pulp worker ‘114363@pulp-api’
migration_service-1 | - ‘core’=‘3.86.0’ with pulp worker ‘114364@pulp-api’
migration_service-1 | - ‘core’=‘3.86.0’ with pulp worker ‘114886@pulp-api’
migration_service-1 | - ‘core’=‘3.86.0’ with pulp worker ‘114885@pulp-api’

Although the error reports that our Pulp setup has workers on version 3.86, that is not the case and Pulp status reports that they are on version indeed 3.88.

#2

Observation:
If we start from a fresh (empty) 3.88 Pulp instance, i.e. no content, remotes, repository, distributions, etc. the upgrade to 3.89 works fine.

But if we have existing content, like our instance, the upgrade would fail.

#3

This is a safeguard to prevent updating the database while too old code is still running.
It appears your database still has heartbeat records from (maybe no longer running) 3.86 api processes.

Are you performing the update in the live system? (zero downtime update [ZDU] / rolling upgrade)
That is what we put these safeguards in for and you should not proceed until replacing all 3.86 processes by 3.87 or better 3.88 . Upon shutdown all the processes are supposed to properly delete their records in the database.

You can also try to clean up manually (This is safe, because it works only if the processes are actually gone):
pulpcore-manager shell -c "from pulpcore.app.models import AppStatus; print(AppStatus.objects.missing().delete())"

#4

We are using the docker compose deployment.

First all the containers are shut down: docker-compose down

We replace the docker images with the latest versions (pulp-minimal and pulp-web) e.g. 3.89.

And then start compose again: docker-compose up

postgres and redis containers are the first to start.

Then it is the migration serivce that faces the error I mentioned, hence the rest of the processes/containers do not start, since they rely on the migration service successful completion.

Now, I got into the migration service container, and ran:

pulpcore-manager shell -c “from pulpcore.app.models import AppStatus; print(AppStatus.objects.missing())”

Result:
<QuerySet []>

Shall I proceed with the delete manual operation you suggested to fix the migration issue?
AppStatus.objects.missing().delete()