Need help with DB migration error

Development
#1

I am working on this (content altering) DB migration: https://github.com/pulp/pulp_deb/pull/760

During this migration, I need to create some new content, to replace existing duplicate content. However, creating new content during the migration results in an error, that I don’t understand. The following is a minimal working example to produce the error (this code is added to the start of the merge_colliding_structure_content() function from the PR):

    # TEST: Try to create a Release content:
    Release = apps.get_model('deb', 'Release')
    Release(distribution='my_test', codename='my_test', suite='my_test').save()

Now running the migration results in the following:

Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/src/pulpcore/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/src/pulp_deb/pulp_deb/app/migrations/0023_merge_colliding_structure_content.py", line 41, in merge_colliding_structure_content
    Release(distribution='my_test', codename='my_test', suite='my_test').save()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 876, in save_base
    parent_inserted = self._save_parents(cls, using, update_fields)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 918, in _save_parents
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1820, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/psycopg/cursor.py", line 723, in execute
    raise ex.with_traceback(None)
django.db.utils.IntegrityError: null value in column "pulp_type" of relation "core_content" violates not-null constraint
DETAIL:  Failing row contains (01884810-740e-74e8-97a5-796aae470f14, 2023-05-23 10:03:37.873655+00, 2023-05-23 10:03:37.87369+00, null, null, 2023-05-23 10:03:37.8737+00, 018847fd-a87f-7e60-bd27-4c4e92e3cc22).

Now I get that for every Release content I create from pulp_deb, pulpcore adds a row to the core_content table, and this has a not null constraint on the pulp_type field. But why is pulpcore failing to correctly initialize this field?

#2

Update: It looks like I don’t get this error if I replace as follows:

from pulp_deb.app.models import Release      # Use regular import
# Release = apps.get_model('deb', 'Release') # Instead of this!

Does anyone know why migrations tend to use the latter? (It is something I copied from existing migrations without questioning it).

#3

I think I can add information here:
Using apps.get_model is definitely the way to go, because this will provide you with the correct historic representation of the database tables at the time, the migration is supposed to run. The version in pulp_deb.app.models may change over time, retrospectively breaking the migration.
OTOH, the whole Master-Detail technology is not part of these “slim” db representations provided in migrations. So it is the duty of the migration writer to properly create those content units with the pulp_type attribute manually set (you cannot rely on any extra methods provided on the model either.).

Either of this should work for you:

Release.objects.create(pulp_type='deb.release', distribution='my_test', codename='my_test', suite='my_test')
Release(pulp_type='deb.release', distribution='my_test', codename='my_test', suite='my_test').save()
2 Likes