Intermittent `500 Internal Server Error`

bli111 · March 27, 2025, 7:00pm

Problem:
500 errors when retrieve content

Expected outcome:

Pulpcore version:

pulp-rpm 3.19.3
pulpcore 3.22.25

Pulp plugins installed and their versions:

Operating system - distribution and version:
Rel 8.10
Other relevant data:

We’ve observed intermittent 500 Internal Server Error responses when retrieving content from our Pulp instance. This issue is isolated to 2 out of our 16 server*, and we have captured traceback logs from the affected instances.

Given that the issue is not occurring cluster-wide and only affects a subset of servers, we are trying to determine whether the root cause is more likely to be network-related, database-related, or potentially something else (e.g., storage, resource exhaustion, etc.).

Could you help us understand what might cause this kind of behavior in a Pulp environment? Would a misconfiguration, DB connectivity issue, or node-specific problem typically result in 500 errors during content retrieval?

Please let us know what additional information we can provide to assist with the investigation.
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 33, in inner
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 259, in cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 237, in _cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._prepare_cursor(self.create_cursor(name))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/utils.py”, line 90, in exit
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: raise dj_exc_value.with_traceback(traceback) from exc_value
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 237, in _cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._prepare_cursor(self.create_cursor(name))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 33, in inner
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]:
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: ret: _R = await loop.run_in_executor(
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/usr/lib64/python3.9/concurrent/futures/thread.py”, line 58, in run
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: result = self.fn(*self.args, **self.kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/asgiref/sync.py”, line 538, in thread_handler
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/content/handler.py”, line 285, in _match_distribution
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: distro_model.objects.select_related(
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, line 431, in get
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: num = len(clone)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, line 262, in len
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: self._fetch_all()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, line 1324, in _fetch_all
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: self._result_cache = list(self._iterable_class(self))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, line 51, in iter
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/sql/compiler.py”, line 1173, in execute_sql
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: cursor = self.connection.cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 33, in inner
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 259, in cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 237, in _cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._prepare_cursor(self.create_cursor(name))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/utils.py”, line 90, in exit
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: raise dj_exc_value.with_traceback(traceback) from exc_value
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 237, in _cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return self._prepare_cursor(self.create_cursor(name))
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 33, in inner
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: return func(*args, **kwargs)
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py”, line 236, in create_cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: cursor = self.connection.cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: django.db.utils.InterfaceError: connection already closed
File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py”, line 236, in create_cursor
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: cursor = self.connection.cursor()
Mar 27 03:50:00 pulpp-ob-994 gunicorn[3172]: django.db.utils.InterfaceError: connection already closed

x9c4 · April 1, 2025, 11:03am

Looks like this is the pulpcore-content app.
We used to have issues with stale django-db connections there. But that was a loooong time ago.
Is it possible, that the load balancer is not distributing fair? Are the processes old when this happens?

bli111 · April 11, 2025, 6:04pm

Yeah, these two hosts usually handle more traffic than the others—it’s always been that way, and there wasn’t any unusual spike when the issue occurred. The processes were old, and we did restart them right after the incident. It hasn’t happened again since, and it hadn’t happened before either. We’re just being cautious in case something changed under the hood that caused it.