Hi Team ,
I have installed openshift pulp operator and as part of proof of concept created local and remote repositories . I am faing an issue while syncing the repo. Sync gets failed in between with Internal server Error and worker nodes gets missing .
For POC , I am using existing pulp repo as remote and source in openshift repo.
Error
[root@yd3248.danskenet.net TEST:~]# pulp rpm repository sync --name local_demo_repo --remote remote_demo_repo
Started background task /pulp/api/v3/tasks/018f956c-d0a6-77d9-928e-6898f1cea96e/
…Traceback (most recent call last):
File “/usr/local/bin/pulp”, line 8, in
sys.exit(main())
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1157, in call
return self.main(*args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1078, in main
rv = self.invoke(ctx)
File “/usr/local/lib/python3.9/site-packages/pulpcore/cli/common/generic.py”, line 289, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python3.9/site-packages/pulpcore/cli/common/generic.py”, line 289, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python3.9/site-packages/pulpcore/cli/common/generic.py”, line 289, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/usr/local/lib/python3.9/site-packages/pulpcore/cli/common/generic.py”, line 289, in invoke
return super().invoke(ctx)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/click/decorators.py”, line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/pulpcore/cli/rpm/repository.py”, line 308, in sync
repository_ctx.sync(body=body)
File “/usr/local/lib/python3.9/site-packages/pulp_glue/rpm/context.py”, line 360, in sync
return super().sync(href, body)
File “/usr/local/lib/python3.9/site-packages/pulp_glue/common/context.py”, line 1266, in sync
return self.call(“sync”, parameters={self.HREF: href or self.pulp_href}, body=body or {})
File “/usr/local/lib/python3.9/site-packages/pulp_glue/common/context.py”, line 722, in call
return self.pulp_ctx.call(
File “/usr/local/lib/python3.9/site-packages/pulp_glue/common/context.py”, line 396, in call
result = self.wait_for_task(result)
File “/usr/local/lib/python3.9/site-packages/pulp_glue/common/context.py”, line 465, in wait_for_task
task = self.api.call(“tasks_read”, parameters={“task_href”: task[“pulp_href”]})
File “/usr/local/lib/python3.9/site-packages/pulp_glue/common/openapi.py”, line 724, in call
response.raise_for_status()
File “/usr/lib/python3.9/site-packages/requests/models.py”, line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://pulp.apps.az3-ost00.danskenet.net/pulp/api/v3/tasks/018f956c-d0a6-77d9-928e-6898f1cea96e/
error from worker pod:
pulp [None]: pulpcore.tasking.entrypoint:INFO: Starting distributed type worker
pulp [None]: pulpcore.tasking.worker:INFO: Cleaning up task 018f956c-d0a6-77d9-928e-6898f1cea96e and marking as failed. Reason: Worker has gone missing.
Workder node logs before getting deleted:
pulp [dfadaedbf8bf40e1b9120bae11bcaddf]: pulpcore.tasking.tasks:INFO: Starting task 018f956c-d0a6-77d9-928e-6898f1cea96e
pulp [dfadaedbf8bf40e1b9120bae11bcaddf]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=local_demo_repo remote=remote_demo_repo
Process Process-1:
Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/django/db/utils.py”, line 98, in inner
return func(*args, **kwargs)
File “/usr/local/lib/python3.9/site-packages/psycopg/server_cursor.py”, line 319, in fetchmany
recs = self._conn.wait(self._fetch_gen(size))
File “/usr/local/lib/python3.9/site-packages/psycopg/connection.py”, line 969, in wait
return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
File “psycopg_binary/_psycopg/waiting.pyx”, line 205, in psycopg_binary._psycopg.wait_c
File “/usr/local/lib/python3.9/site-packages/psycopg/server_cursor.py”, line 173, in _fetch_gen
res = yield from self._conn._exec_command(query, result_format=self._format)
File “/usr/local/lib/python3.9/site-packages/psycopg/connection.py”, line 469, in _exec_command
result = (yield from execute(self.pgconn))[-1]
File “psycopg_binary/_psycopg/generators.pyx”, line 83, in execute
File “psycopg_binary/_psycopg/generators.pyx”, line 133, in fetch_many
File “psycopg_binary/_psycopg/generators.pyx”, line 181, in fetch
psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
database pods has below error
2024-05-20 10:14:01.995 UTC [1] LOG: server process (PID 596310) was terminated by signal 9: Killed
2024-05-20 10:14:01.995 UTC [1] DETAIL: Failed process was running: FETCH FORWARD 2000 FROM “_django_curs_140605686543936_sync_34”
2024-05-20 10:14:01.995 UTC [1] LOG: terminating any other active server processes
2024-05-20 10:14:01.995 UTC [600223] WARNING: terminating connection because of crash of another server process
2024-05-20 10:14:01.995 UTC [600223] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Any suggestions ?