Problem:
Hi,
some of my pulpcore-workers are not running anymore.
They stay in constant state:
pulpcore-worker@1.service loaded activating auto-restart Pulp Worker
pulpcore-worker@2.service loaded activating auto-restart Pulp Worker
pulp status does not show running workers anymore:
{
“versions”: [
{
“component”: “core”,
“version”: “3.22.4”,
“package”: “pulpcore”
},
{
“component”: “rpm”,
“version”: “3.19.4”,
“package”: “pulp-rpm”
}
],
“online_workers”: [],
“online_content_apps”: [],
“database_connection”: {
“connected”: true
},
“redis_connection”: {
“connected”: true
},
“storage”: {
“total”: 193179156480,
“used”: 110952255488,
“free”: 82226900992
},
“content_settings”: {
“content_origin”: “pulp3.lnxsrv.abraxas.ch”,
“content_path_prefix”: “/pulp/repos/”
And the log is filling:
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: DETAIL: Failing row contains (3bbcf542-fc8b-43eb-91bd-5a05a89b6a66, 2023-04-28 12:30:06.479253+00, 2023-04-28 12:30:06.479274+00, 42208@pulp3.lnxsrv.abraxas.ch, 2023-04-28 12:30:06.479286+00, null).
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: The above exception was the direct cause of the following exception:
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: Traceback (most recent call last):
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/bin/pulpcore-worker”, line 8, in
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: sys.exit(worker())
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/click/core.py”, line 1130, in call
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return self.main(*args, **kwargs)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/click/core.py”, line 1055, in main
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: rv = self.invoke(ctx)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/click/core.py”, line 1404, in invoke
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return ctx.invoke(self.callback, **ctx.params)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/click/core.py”, line 760, in invoke
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return __callback(*args, **kwargs)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/entrypoint.py”, line 28, in worker
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: NewPulpWorker().run_forever()
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py”, line 128, in init
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: self.worker = handle_worker_heartbeat(self.name)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py”, line 104, in handle_worker_heartbeat
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: worker, created = Worker.objects.get_or_create(name=worker_name)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py”, line 85, in manager_method
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py”, line 588, in get_or_create
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return self.create(**params), True
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py”, line 453, in create
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: obj.save(force_insert=True, using=self.db)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/lib64/python3.9/contextlib.py”, line 79, in inner
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return func(*args, **kwds)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django_lifecycle/mixins.py”, line 169, in save
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: save(*args, **kwargs)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/base.py”, line 739, in save
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: self.save_base(using=using, force_insert=force_insert,
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/base.py”, line 776, in save_base
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: updated = self._save_table(
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/base.py”, line 881, in _save_table
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/base.py”, line 919, in _do_insert
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return manager._insert(
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/manager.py”, line 85, in manager_method
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/query.py”, line 1270, in _insert
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return query.get_compiler(using=using).execute_sql(returning_fields)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/models/sql/compiler.py”, line 1416, in execute_sql
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: cursor.execute(sql, params)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 66, in execute
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 75, in _execute_with_wrappers
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return executor(sql, params, many, context)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 84, in _execute
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return self.cursor.execute(sql, params)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/utils.py”, line 90, in exit
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: raise dj_exc_value.with_traceback(traceback) from exc_value
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: File “/usr/local/lib/pulp/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 84, in _execute
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: return self.cursor.execute(sql, params)
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: django.db.utils.IntegrityError: null value in column “versions” of relation “core_worker” violates not-null constraint
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: DETAIL: Failing row contains (3bbcf542-fc8b-43eb-91bd-5a05a89b6a66, 2023-04-28 12:30:06.479253+00, 2023-04-28 12:30:06.479274+00, 42208@pulp3.lnxsrv.abraxas.ch, 2023-04-28 12:30:06.479286+00, null).
Expected outcome:
Start of all services and pulp to work as expected
Pulpcore version:
“versions”: [
{
“component”: “core”,
“version”: “3.22.4”,
“package”: “pulpcore”
},
Pulp plugins installed and their versions:
{
“component”: “rpm”,
“version”: “3.19.4”,
“package”: “pulp-rpm”
}
Operating system - distribution and version:
RHEL9 Server
Other relevant data:
This happened after i run the pulp_installer.
I was trying to fix an issue with the following setting:
Changed:
content_path_prefix: “/pulp/repos”
to
content_path_prefix: “/pulp/repos/”
I did not see any erros in the installer output. Only that the workers are not running.
It would be great if someone has ay idea how to find out what is missing here!
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: django.db.utils.IntegrityError: null value in column “versions” of relation “core_worker” violates not-null constraint
Apr 28 14:30:06 pulp3 pulpcore-worker[42208]: DETAIL: Failing row contains (3bbcf542-fc8b-43eb-91bd-5a05a89b6a66, 2023-04-28 12:30:06.479253+00, 2023-04-28 12:30:06.479274+00,
Greetz
Andreas