Frequent pulp worker crash with error django.db.utils.OperationalError: connection timeout expired

**Problem:**We have observed on multiple occasions that the Pulp server becomes unresponsive, even though the Docker container appears to be running. Upon reviewing the logs, we frequently encounter the following errors:

File “/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py”, line 256, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/django/db/backends/postgresql/base.py”, line 332, in get_new_connection
connection = self.Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/psycopg/connection.py”, line 122, in connect
raise last_ex.with_traceback(None)
django.db.utils.OperationalError: connection timeout expired
connection timeout expired
[2026-06-16 05:58:10 +0000] [44602] [INFO] Worker exiting (pid: 44602)
[2026-06-16 05:58:11 +0000] [44583] [ERROR] Worker (pid:44597) exited with code 3.
[2026-06-16 05:58:11 +0000] [44583] [ERROR] Worker (pid:44602) exited with code 3.

We suspect that the broken connection to the database might be caused by a few unreachable remote encrypted data entries in the database. It appears that all files in the /settings/certs directory were inadvertently regenerated, including the database_fields.symmetric.key file. As a result, all previously configured remote Git repositories with encrypted fields in the database have become unreachable.

We attempted to delete these remote entries directly using the following Python command: Remote.objects…filter(pk=’{remote_pk}’).delete()
However, this approach consistently results in an error

“/usr/local/lib64/python3.11/site-packages/cryptography/fernet.py”, line 206, in decrypt raise InvalidToken cryptography.fernet.InvalidToken

Could you please confirm if the frequent worker crashes we are experiencing could be related to the modification of the database_fields.symmetric.key file? Additionally, could you advise on the best way to clean up all unreachable entries directly from the database (by sql query) as we have the pulp_href list of the three remote Git repositories that are affected by this issue.

Expected outcome:

Pulpcore version: 3.113.0

Pulp plugins installed and their versions:

Operating system - distribution and version: pulp docker image pulp/pulp:3.113

Other relevant data: