Task - failed: 'null value in column "pulp_labels" of relation "core_content" violates not-null constraint

Support
#1

Problem:

Please assist.

During publication.

Task /pulp/api/v3/tasks/01956a92-e6ee-72ee-b375-377087b96079/ failed: 'null value in column "pulp_labels" of relation "core_content" violates not-null constraint

[1740](https://atc-github.azure.c.. etc) DETAIL: Failing row contains (01956a9b-3a73-75da-881f-002f1e5c5d8f, 2025-03-06 08:38:32.821695+00, 2025-03-06 08:38:32.821713+00, core.publishedmetadata, null, 2025-03-06 08:38:32.821727+00, 019513aa-8ee7-77d4-af49-eb445030ae0b, null).'

    - name: Create rpm Publication of Repositories
      pulp.squeezer.rpm_publication:
         pulp_url: "{{ PULP_SVR }}"
         repository: "{{ item.name }}"
         validate_certs: false
         timeout: 600
         state: present
      register: pub_result
      loop: "{{ repo_data.Repository }}"
      loop_control:
         index_var: p_idx

Expected outcome:

To not have this error.

Pulpcore version:

    {
      "component": "core",
      "version": "3.68.1",
      "package": "pulpcore",
      "module": "pulpcore.app",
      "domain_compatible": true
    },

Pulp plugins installed and their versions:

Collection                               Version
---------------------------------------- -------
pulp.squeezer                            0.1.0

Package            Version
------------------ ---------
ansible            11.2.0
ansible-core       2.18.2
certifi            2025.1.31
cffi               1.17.1
charset-normalizer 3.4.1
click              8.1.8
cryptography       44.0.1
idna               3.10
Jinja2             3.1.5
jmespath           1.0.1
MarkupSafe         3.0.2
multidict          6.1.0
packaging          24.2
pip                25.0.1
pulp-cli           0.30.2
pulp-glue          0.30.2
pycparser          2.22
PyYAML             6.0.2
requests           2.32.3
resolvelib         1.0.1
schema             0.7.7
setuptools         65.5.0
tomli_w            1.2.0
urllib3            2.3.0

Operating system - distribution and version:

Other relevant data:

2025-03-06T08:30:00.107062348+01:00 stderr F pulp [76ac671b7b434fea87474124a6fbdbb5]: pulpcore.tasking.tasks:INFO: Starting task 01956a5c-7878-776d-a278-a5bebe525423 in domain: default
2025-03-06T08:30:00.143516236+01:00 stderr F pulp [76ac671b7b434fea87474124a6fbdbb5]: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=active-P-2025Q02.00-ABC-SLES12-SP4-latest, version=102
2025-03-06T08:31:20.406590328+01:00 stderr F pulp [76ac671b7b434fea87474124a6fbdbb5]: pulpcore.tasking.tasks:INFO: Task[pulp_rpm.app.tasks.publishing.publish] 01956a5c-7878-776d-a278-a5bebe525423 failed (IntegrityError: null value in column “pulp_labels” of relation “core_content” violates not-null constraint
2025-03-06T08:31:20.406590328+01:00 stderr F DETAIL: Failing row contains (01956a5d-ad7d-726d-890e-a03feadd0680, 2025-03-06 07:31:19.039538+00, 2025-03-06 07:31:19.039556+00, core.publishedmetadata, null, 2025-03-06 07:31:19.039569+00, 019513aa-8ee7-77d4-af49-eb445030ae0b, null).) in domain: default
2025-03-06T08:31:20.406815211+01:00 stderr F pulp [76ac671b7b434fea87474124a6fbdbb5]: pulpcore.tasking.tasks:INFO: File “/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py”, line 68, in _execute_task
2025-03-06T08:31:20.406815211+01:00 stderr F result = func(*args, **kwargs)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py”, line 385, in publish
2025-03-06T08:31:20.406815211+01:00 stderr F generate_repo_metadata(
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py”, line 683, in generate_repo_metadata
2025-03-06T08:31:20.406815211+01:00 stderr F PublishedMetadata.create_from_file(
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/pulpcore/app/models/publication.py”, line 303, in create_from_file
2025-03-06T08:31:20.406815211+01:00 stderr F content.save()
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/lib64/python3.9/contextlib.py”, line 79, in inner
2025-03-06T08:31:20.406815211+01:00 stderr F return func(*args, **kwds)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django_lifecycle/mixins.py”, line 139, in save
2025-03-06T08:31:20.406815211+01:00 stderr F save(*args, **kwargs)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/models/base.py”, line 814, in save
2025-03-06T08:31:20.406815211+01:00 stderr F self.save_base(
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/models/base.py”, line 876, in save_base
2025-03-06T08:31:20.406815211+01:00 stderr F parent_inserted = self._save_parents(cls, using, update_fields)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/models/base.py”, line 918, in _save_parents
2025-03-06T08:31:20.406815211+01:00 stderr F updated = self._save_table(
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/models/base.py”, line 1020, in _save_table
2025-03-06T08:31:20.406815211+01:00 stderr F results = self._do_insert(
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/models/base.py”, line 1061, in _do_insert
2025-03-06T08:31:20.406815211+01:00 stderr F return manager._insert(
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/models/manager.py”, line 87, in manager_method
2025-03-06T08:31:20.406815211+01:00 stderr F return getattr(self.get_queryset(), name)(*args, **kwargs)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/models/query.py”, line 1805, in _insert
2025-03-06T08:31:20.406815211+01:00 stderr F return query.get_compiler(using=using).execute_sql(returning_fields)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py”, line 1822, in execute_sql
2025-03-06T08:31:20.406815211+01:00 stderr F cursor.execute(sql, params)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py”, line 67, in execute
2025-03-06T08:31:20.406815211+01:00 stderr F return self._execute_with_wrappers(
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py”, line 80, in _execute_with_wrappers
2025-03-06T08:31:20.406815211+01:00 stderr F return executor(sql, params, many, context)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py”, line 89, in _execute
2025-03-06T08:31:20.406815211+01:00 stderr F return self.cursor.execute(sql, params)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/utils.py”, line 91, in exit
2025-03-06T08:31:20.406815211+01:00 stderr F raise dj_exc_value.with_traceback(traceback) from exc_value
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py”, line 89, in _execute
2025-03-06T08:31:20.406815211+01:00 stderr F return self.cursor.execute(sql, params)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:31:20.406815211+01:00 stderr F File “/usr/local/lib/python3.9/site-packages/psycopg/cursor.py”, line 97, in execute
2025-03-06T08:31:20.406815211+01:00 stderr F raise ex.with_traceback(None)
2025-03-06T08:31:20.406815211+01:00 stderr F
2025-03-06T08:36:29.775514698+01:00 stderr F pulp [40023436165d497e84f8b4fb2b01f736]: pulpcore.tasking.tasks:INFO: Starting task 01956a62-6a95-70fb-b7ae-3102cdf0b5a5 in domain: default
2025-03-06T08:36:29.793378632+01:00 stderr F pulp [40023436165d497e84f8b4fb2b01f736]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=upstream-SLE15-SP5-Installer-Updates remote=remote_upstream-SLE15-SP5-Installer-Updates
2025-03-06T08:36:29.870879022+01:00 stderr F pulp [40023436165d497e84f8b4fb2b01f736]: pulpcore.tasking.tasks:INFO: Task completed 01956a62-6a95-70fb-b7ae-3102cdf0b5a5 in domain: default

pulp task show --href /pulp/api/v3/tasks/01956b61-a424-7e08-acfa-ee3a3d4fef14/

{
  "pulp_href": "/pulp/api/v3/tasks/01956b61-a424-7e08-acfa-ee3a3d4fef14/",
  "prn": "prn:core.task:01956b61-a424-7e08-acfa-ee3a3d4fef14",
  "pulp_created": "2025-03-06T12:15:16.005429Z",
  "pulp_last_updated": "2025-03-06T12:15:16.005445Z",
  "state": "failed",
  "name": "pulp_rpm.app.tasks.publishing.publish",
  "logging_cid": "3257c986c6b34cefbacbe3ba1f9bde93",
  "created_by": "/pulp/api/v3/users/1/",
  "unblocked_at": "2025-03-06T12:15:16.032655Z",
  "started_at": "2025-03-06T12:15:16.126058Z",
  "finished_at": "2025-03-06T12:16:46.474406Z",
  "error": {
    "traceback": "  File \"/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py\", line 68, in _execute_task\n    result = func(*args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py\", line 385, in publish\n    generate_repo_metadata(\n  File \"/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py\", line 683, in generate_repo_metadata\n    PublishedMetadata.create_from_file(\n  File \"/usr/local/lib/python3.9/site-packages/pulpcore/app/models/publication.py\", line 303, in create_from_file\n    content.save()\n  File \"/usr/lib64/python3.9/contextlib.py\", line 79, in inner\n    return func(*args, **kwds)\n  File \"/usr/local/lib/python3.9/site-packages/django_lifecycle/mixins.py\", line 139, in save\n    save(*args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/base.py\", line 814, in save\n    self.save_base(\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/base.py\", line 876, in save_base\n    parent_inserted = self._save_parents(cls, using, update_fields)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/base.py\", line 918, in _save_parents\n    updated = self._save_table(\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/base.py\", line 1020, in _save_table\n    results = self._do_insert(\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/base.py\", line 1061, in _do_insert\n    return manager._insert(\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/manager.py\", line 87, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/query.py\", line 1805, in _insert\n    return query.get_compiler(using=using).execute_sql(returning_fields)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1822, in execute_sql\n    cursor.execute(sql, params)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(\n  File \"/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py\", line 80, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py\", line 89, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/utils.py\", line 91, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py\", line 89, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/usr/local/lib/python3.9/site-packages/psycopg/cursor.py\", line 97, in execute\n    raise ex.with_traceback(None)\n",
    "description": "null value in column \"pulp_labels\" of relation \"core_content\" violates not-null constraint\nDETAIL:  Failing row contains (01956b62-ff6f-7f93-9088-19deff1aae02, 2025-03-06 12:16:44.915212+00, 2025-03-06 12:16:44.915229+00, core.publishedmetadata, null, 2025-03-06 12:16:44.915241+00, 019513aa-8ee7-77d4-af49-eb445030ae0b, null)."
  },
  "worker": "/pulp/api/v3/workers/01956a12-8d2b-7cab-85f1-8d0ccb0cb492/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Generating repository metadata",
      "code": "publish.generating_metadata",
      "state": "failed",
      "total": 1,
      "done": 0,
      "suffix": null
    }
  ],
  "created_resources": [],
  "reserved_resources_record": [
    "shared:prn:rpm.rpmrepository:01951945-fc4f-792c-bcaf-bf57bfb75418",
    "shared:prn:core.domain:019513aa-8ee7-77d4-af49-eb445030ae0b"
  ]
}

Thank you.

1 Like
#2

What version of pulp_rpm is running? Also, Content.pulp_labels was introduced in core/3.73 - based on the output you’ve shared, your installation’s python code doesn’t match the database.

#3

Hi

Because of this:

https://discourse.pulpproject.org/t/boto3-botocore-a-header-you-provided-implies-functionality-that-is-not-implemented-transfer-encoding/1900/10

We cannot use 3.73, till our NetApp is upgraded.

But I tried 3.73 briefly this morning. But reversed because of S3 error

Did this hurt the PG database?

Pulp-rpm, is this the one you are referring to:
“version”: “3.27.2”,
“package”: “pulp-rpm”

  "versions": [
    {
      "component": "core",
      "version": "3.68.1",
      "package": "pulpcore",
      "module": "pulpcore.app",
      "domain_compatible": true
    },
    {
      "component": "ansible",
      "version": "0.23.1",
      "package": "pulp-ansible",
      "module": "pulp_ansible.app",
      "domain_compatible": false
    },
    {
      "component": "container",
      "version": "2.22.0",
      "package": "pulp-container",
      "module": "pulp_container.app",
      "domain_compatible": false
    },
    {
      "component": "deb",
      "version": "3.4.0",
      "package": "pulp_deb",
      "module": "pulp_deb.app",
      "domain_compatible": false
    },
    {
      "component": "maven",
      "version": "0.8.1",
      "package": "pulp-maven",
      "module": "pulp_maven.app",
      "domain_compatible": false
    },
    {
      "component": "ostree",
      "version": "2.4.4",
      "package": "pulp-ostree",
      "module": "pulp_ostree.app",
      "domain_compatible": true
    },
    {
      "component": "python",
      "version": "3.12.5",
      "package": "pulp-python",
      "module": "pulp_python.app",
      "domain_compatible": true
    },
    {
      "component": "rpm",
      "version": "3.27.2",
      "package": "pulp-rpm",
      "module": "pulp_rpm.app",
      "domain_compatible": true
    },
    {
      "component": "certguard",
      "version": "3.68.1",
      "package": "pulpcore",
      "module": "pulp_certguard.app",
      "domain_compatible": true
    },
    {
      "component": "file",
      "version": "3.68.1",
      "package": "pulpcore",
      "module": "pulp_file.app",
      "domain_compatible": true
    }
  ]
#4

Well, the database is now upgraded to core/3.73, but the python code isn’t, and that’s why you’re getting “impossible errors” from the app :frowning: You can rollback Django migrations - see Migrations | Django documentation | Django. I haven’t done that myself, mind, and I don’t know for sure that the core/3.73 migrations are reversible (those are 0129_content_pulp_labels and 0130_upstreampulp_policy. rpm/3.28 didn’t add any migrations.

1 Like
#5

Understood.
Thank you.

I’ll try, else start fresh.
Nice lesson. :wink:

1 Like
#6

Good luck - and do let me know how it goes, I prob should do some experimenting myself just to have a feel for how that process goes! :slight_smile:

#7

The rollback did work 100%.

On pulp-minimal image 3.68:

/usr/local/bin/pulpcore-manager showmigrations

Under core.

 [X] 0021_remove_release_from_structure_types
 [X] 0022_alter_aptdistribution_distribution_ptr_and_more
 [X] 0023_add_default_signing_services
 [X] 0024_add_release_fields
 [X] 0025_merge_colliding_structure_content

On image 3.73

Under core:

 [X] 0021_remove_release_from_structure_types
 [X] 0022_alter_aptdistribution_distribution_ptr_and_more
 [X] 0023_add_default_signing_services
 [X] 0024_add_release_fields
 [X] 0025_merge_colliding_structure_content
 [X] 0026_aptrepository_publish_upstream_release_fields
 [X] 0027_alter_aptpublication_structured
 [X] 0028_sourcepackage_sourcepackagereleasecomponent_and_more
 [X] 0029_distributedpublication

Start pulp-minimal 3.73 and execute:

python3 /usr/local/lib/python3.9/site-packages/pulpcore/app/manage.py migrate core 0126

This rolls back to 0126

/usr/local/bin/pulpcore-manager showmigrations

 [X] 0122_record_last_replication_timestamp
 [X] 0123_upstreampulp_q_select
 [X] 0124_task_deferred_task_immediate
 [X] 0125_openpgpdistribution_openpgpkeyring_openpgppublickey_and_more
 [X] 0126_remoteartifact_failed_at
 [ ] 0127_remove_upstreampulp_pulp_label_select
 [ ] 0128_domain_pulp_labels
 [ ] 0129_content_pulp_labels
 [ ] 0130_upstreampulp_policy

Start pulp-minimal in version 3.68
And the repository builds is successful again.

Should have rolled back to 0125 but 0126 does not hurt.

Thank you for help and experience.

3 Likes
#8

Outstanding! Thank you for a thorough after-action report too - that will help anybody else who might need to do this in the future (like, say, me, trying to help someone in six months… :slight_smile: )

Glad you’re up and running again

1 Like