Problem: Hello, posted this on the foreman discourse but no answer yet. Any advice would be greatly appreciated!
Pulp workers are constantly failing and restarting meaning syncs of my repos are failing. I am pretty new to foreman so not sure what is going on. From /var/log/messages the main lines I think it’s complaining about are:
Oct 19 09:02:26 foreman pulpcore-worker-8: Traceback (most recent call last):
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/utils.py”, line 84, in _execute
Oct 19 09:02:26 foreman pulpcore-worker-8: return self.cursor.execute(sql, params)
Oct 19 09:02:26 foreman pulpcore-worker-8: psycopg2.errors.UndefinedTable: relation “core_taskschedule” does not exist
Oct 19 09:02:26 foreman pulpcore-worker-8: LINE 1: …sk_name", “core_taskschedule”.“last_task_id” FROM "core_task…
Oct 19 09:02:26 foreman pulpcore-worker-8: ^
Expected outcome: Repo’s sync successfully.
Pulpcore version: tfm-pulpcore-python3-pulp-python-3.7.1-1 (I’m not exactly sure how to tell)
Pulp plugins installed and their versions: Lots… seems the servers has lots of pulp rpms installed.
Operating system - distribution and version: Centos 7.9
Other relevant data:
Hopefully the above has all the data needed. I’m not sure what is going on here so any advice would be appreciated. Just in case here is the full log:
Oct 19 09:02:26 foreman pulpcore-worker-8: Traceback (most recent call last):
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/utils.py”, line 84, in _execute
Oct 19 09:02:26 foreman pulpcore-worker-8: return self.cursor.execute(sql, params)
Oct 19 09:02:26 foreman pulpcore-worker-8: psycopg2.errors.UndefinedTable: relation “core_taskschedule” does not exist
Oct 19 09:02:26 foreman pulpcore-worker-8: LINE 1: …sk_name", “core_taskschedule”.“last_task_id” FROM “core_task…
Oct 19 09:02:26 foreman pulpcore-worker-8: ^
Oct 19 09:02:26 foreman pulpcore-worker-8: The above exception was the direct cause of the following exception:
Oct 19 09:02:26 foreman pulpcore-worker-8: Traceback (most recent call last):
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/bin/pulpcore-worker”, line 11, in
Oct 19 09:02:26 foreman pulpcore-worker-8: load_entry_point(‘pulpcore==3.18.10’, ‘console_scripts’, ‘pulpcore-worker’)()
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/click/core.py”, line 1128, in call
Oct 19 09:02:26 foreman pulpcore-worker-8: return self.main(*args, **kwargs)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/click/core.py”, line 1053, in main
Oct 19 09:02:26 foreman pulpcore-worker-8: rv = self.invoke(ctx)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/click/core.py”, line 1395, in invoke
Oct 19 09:02:26 foreman pulpcore-worker-8: return ctx.invoke(self.callback, **ctx.params)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/click/core.py”, line 754, in invoke
Oct 19 09:02:26 foreman pulpcore-worker-8: return __callback(*args, **kwargs)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/entrypoint.py”, line 28, in worker
Oct 19 09:02:26 foreman pulpcore-worker-8: NewPulpWorker().run_forever()
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py”, line 368, in run_forever
Oct 19 09:02:26 foreman pulpcore-worker-8: self.sleep()
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py”, line 280, in sleep
Oct 19 09:02:26 foreman pulpcore-worker-8: self.beat()
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py”, line 153, in beat
Oct 19 09:02:26 foreman pulpcore-worker-8: dispatch_scheduled_tasks()
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/tasking/tasks.py”, line 104, in dispatch_scheduled_tasks
Oct 19 09:02:26 foreman pulpcore-worker-8: for task_schedule in TaskSchedule.objects.filter(next_dispatch__lte=now).filter(
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py”, line 280, in iter
Oct 19 09:02:26 foreman pulpcore-worker-8: self._fetch_all()
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py”, line 1324, in _fetch_all
Oct 19 09:02:26 foreman pulpcore-worker-8: self._result_cache = list(self._iterable_class(self))
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py”, line 51, in iter
Oct 19 09:02:26 foreman pulpcore-worker-8: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py”, line 1175, in execute_sql
Oct 19 09:02:26 foreman pulpcore-worker-8: cursor.execute(sql, params)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/utils.py”, line 66, in execute
Oct 19 09:02:26 foreman pulpcore-worker-8: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/utils.py”, line 75, in _execute_with_wrappers
Oct 19 09:02:26 foreman pulpcore-worker-8: return executor(sql, params, many, context)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/utils.py”, line 84, in _execute
Oct 19 09:02:26 foreman pulpcore-worker-8: return self.cursor.execute(sql, params)
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/utils.py”, line 90, in exit
Oct 19 09:02:26 foreman pulpcore-worker-8: raise dj_exc_value.with_traceback(traceback) from exc_value
Oct 19 09:02:26 foreman pulpcore-worker-8: File “/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/backends/utils.py”, line 84, in _execute
Oct 19 09:02:26 foreman pulpcore-worker-8: return self.cursor.execute(sql, params)
Oct 19 09:02:26 foreman pulpcore-worker-8: django.db.utils.ProgrammingError: relation “core_taskschedule” does not exist
Oct 19 09:02:26 foreman pulpcore-worker-8: LINE 1: …sk_name”, “core_taskschedule”.“last_task_id” FROM "core_task…
Oct 19 09:02:26 foreman pulpcore-worker-8: ^
Oct 19 09:02:27 foreman systemd: pulpcore-worker@8.service: main process exited, code=exited, status=1/FAILURE
Oct 19 09:02:27 foreman systemd: Unit pulpcore-worker@8.service entered failed state.
Oct 19 09:02:27 foreman systemd: pulpcore-worker@8.service failed.