Problem:
500 errors when retrieve content
Expected outcome:
Pulpcore version:
pulp-rpm 3.19.3
pulpcore 3.22.25
Pulp plugins installed and their versions:
Operating system - distribution and version:
Rel 8.10
Other relevant data:
We’ve observed intermittent 500 Internal Server Error responses when retrieving content from our Pulp instance. This issue is isolated to 2 out of our 16 server*, and we have captured traceback logs from the affected instances.
Given that the issue is not occurring cluster-wide and only affects a subset of servers, we are trying to determine whether the root cause is more likely to be network-related, database-related, or potentially something else (e.g., storage, resource exhaustion, etc.).
Could you help us understand what might cause this kind of behavior in a Pulp environment? Would a misconfiguration, DB connectivity issue, or node-specific problem typically result in 500 errors during content retrieval?
Please let us know what additional information we can provide to assist with the investigation.
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 33, in inner
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 259, in cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 237, in _cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._prepare_cursor(self.create_cursor(name))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/utils.py”, line 90, in exit
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: raise dj_exc_value.with_traceback(traceback) from exc_value
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 237, in _cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._prepare_cursor(self.create_cursor(name))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 33, in inner
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]:
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: ret: _R = await loop.run_in_executor(
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/usr/lib64/python3.9/concurrent/futures/thread.py”, line 58, in run
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: result = self.fn(*self.args, **self.kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/asgiref/sync.py”, line 538, in thread_handler
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/content/handler.py”, line 285, in _match_distribution
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: distro_model.objects.select_related(
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, line 431, in get
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: num = len(clone)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, line 262, in len
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: self._fetch_all()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, line 1324, in _fetch_all
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: self._result_cache = list(self._iterable_class(self))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, line 51, in iter
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/sql/compiler.py”, line 1173, in execute_sql
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: cursor = self.connection.cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 33, in inner
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 259, in cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 237, in _cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._prepare_cursor(self.create_cursor(name))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/utils.py”, line 90, in exit
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: raise dj_exc_value.with_traceback(traceback) from exc_value
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 237, in _cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._prepare_cursor(self.create_cursor(name))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 33, in inner
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py”, line 236, in create_cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: cursor = self.connection.cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: django.db.utils.InterfaceError: connection already closed
File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py”, line 236, in create_cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: cursor = self.connection.cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: django.db.utils.InterfaceError: connection already closed