Getting 500 error after upgrade from 3.22.4 to 3.22.21

bli111 · January 31, 2024, 6:14pm

Problem:
Api returns 500 error.

Expected outcome:

Pulpcore version:
pulpcore 3.22.21

Pulp plugins installed and their versions:

Operating system - distribution and version:
pulp-ansible 0.16.1
pulp-cli 0.23.1
pulp-deb 2.20.2
pulp-file 1.12.0
pulp-glue 0.23.1
pulp-rpm 3.19.3
pulpcore 3.22.21
django-currentuser 0.5.3
django-filter 22.1
django-guid 3.3.0
django-import-export 3.3.6
django-lifecycle 1.0.0
django-readonly-field 1.1.2
django-storages 1.14.2
djangorestframework 3.14.0
djangorestframework-queryfields 1.0.0
Other relevant data:
We are currently running 3.22.4 with external postgres, redis and web authentication. We added a new physical host running api, content and worker instances. The new instances share the same postgres and redis. The pulp_installer picked up the latest version for 3.22.21. The new api instance return 500 error and we got the errors below. We tried to downgrade the django to match the version that we are currently running and it didn’t work. Please advise how this can be fixed.

Current Pulp Instance:
pulp-ansible 0.16.1
pulp-cli 0.19.0
pulp-deb 2.20.2
pulp-file 1.12.0
pulp-glue 0.19.0
pulp-rpm 3.19.3
pulp-rpm-client 3.19.3
pulpcore 3.22.4
pulpcore-client 3.22.4
django-currentuser 0.5.3
django-extensions 3.2.3
django-filter 22.1
django-guid 3.3.0
django-import-export 3.0.2
django-lifecycle 1.0.0
django-readonly-field 1.1.2
django-storages 1.13.2
djangorestframework 3.13.1
djangorestframework-queryfields 1.0.0

New instance:
pulp-ansible 0.16.1
pulp-cli 0.23.1
pulp-deb 2.20.2
pulp-file 1.12.0
pulp-glue 0.23.1
pulp-rpm 3.19.3
pulpcore 3.22.21
django-currentuser 0.5.3
django-filter 22.1
django-guid 3.3.0
django-import-export 3.3.6
django-lifecycle 1.0.0
django-readonly-field 1.1.2
django-storages 1.14.2
djangorestframework 3.14.0
djangorestframework-queryfields 1.0.0

<an 31 12:41:40 pulphost2 gunicorn[72949]: pulp [ae6a778784124c96bf28de9f02936bc5]: django.request:ERROR: Internal Server Error: /pulp/api>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: Traceback (most recent call last):
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/cryptography/fernet.py”, line >
Jan 31 12:41:40 pulphost2 gunicorn[72949]: h.verify(data[-32:])
Jan 31 12:41:40 pulphost2 gunicorn[72949]: cryptography.exceptions.InvalidSignature: Signature did not match digest.
Jan 31 12:41:40 pulphost2 gunicorn[72949]: During handling of the above exception, another exception occurred:
Jan 31 12:41:40 pulphost2 gunicorn[72949]: Traceback (most recent call last):
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File "/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/core/handlers/exception>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: response = get_response(request)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/core/handlers/base.py”,>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File "/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/views/decorators/csrf.p>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return view_func(*args, **kwargs)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/viewsets.py”, l>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return self.dispatch(request, *args, **kwargs)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/views.py”, line>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: response = self.handle_exception(exc)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/views.py”, line>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: self.raise_uncaught_exception(exc)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/views.py”, line>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: raise exc
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/views.py”, line>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: response = handler(request, *args, **kwargs)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/mixins.py”, lin>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return self.get_paginated_response(serializer.data)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/serializers.py”>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: ret = super().data
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/serializers.py”>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: self._data = self.to_representation(self.instance)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/serializers.py”>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return [
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/serializers.py”>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: self.child.to_representation(item) for item in iterable
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/serializers.py”>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: attribute = field.get_attribute(instance)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/relations.py”, >
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return super().get_attribute(instance)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/fields.py”, lin>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return get_attribute(instance, self.source_attrs)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/rest_framework/fields.py”, lin>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: instance = getattr(instance, attr)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File "/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/fields/relate>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: rel_obj = self.get_object(instance)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File "/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/fields/relate>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return qs.get(self.field.get_reverse_related_filter(instance))
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, li>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: num = len(clone)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, li>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: self._fetch_all()
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, li>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: self._result_cache = list(self._iterable_class(self))
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py”, li>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: for row in compiler.results_iter(results):
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File "/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/sql/compiler.>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: value = converter(value, expression, connection)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/app/models/fields.py”>
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return force_str(self._fernet.decrypt(force_bytes(value)))
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/cryptography/fernet.py”, line >
Jan 31 12:41:40 pulphost2 gunicorn[72949]: return self._decrypt_data(data, timestamp, time_info)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/cryptography/fernet.py”, line >
Jan 31 12:41:40 pulphost2 gunicorn[72949]: self._verify_signature(data)
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/cryptography/fernet.py”, line >
Jan 31 12:41:40 pulphost2 gunicorn[72949]: raise InvalidToken
Jan 31 12:41:40 pulphost2 gunicorn[72949]: cryptography.fernet.InvalidToken
→

ggainey · February 1, 2024, 6:13pm

The base error is :

Jan 31 12:41:40 pulphost2 gunicorn[72949]: Traceback (most recent call last):
Jan 31 12:41:40 pulphost2 gunicorn[72949]: File “/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/cryptography/fernet.py”, line >
Jan 31 12:41:40 pulphost2 gunicorn[72949]: h.verify(data[-32:])
Jan 31 12:41:40 pulphost2 gunicorn[72949]: cryptography.exceptions.InvalidSignature: Signature did not match digest.

This appears to be happening way before any pulp-code gets involved. You say there are two systems involved, sharing services/db - have you checked that the times are the same for everyone? Mismatched timestamps can def cause Fun Problems w/ crypto verification, for example.

bli111 · February 2, 2024, 5:45pm

Thanks Grant. We use the chronyc to synchronize the time for all our servers. The time synchronization is not an issue.

Reference ID : 0A0A0A0C (timeserver)
Stratum : 3
Ref time (UTC) : Fri Feb 02 16:54:01 2024
System time : 0.000004756 seconds fast of NTP time
Last offset : -0.000000367 seconds
RMS offset : 0.000009729 seconds
Frequency : 1.706 ppm fast
Residual freq : -0.000 ppm
Skew : 0.027 ppm
Root delay : 0.001693913 seconds
Root dispersion : 0.001246301 seconds
Update interval : 64.3 seconds
Leap status : Normal

The old server which is running 3.22.4 continues to work. The new server running 3.22.21 has this issue. We have exact the same settings.py on both server and both sharing the same database. We specify webserver authentication in the settings:
“AUTHENTICATION_BACKENDS = [‘pulpcore.app.authentication.PulpNoCreateRemoteUserBackend’]
REST_FRAMEWORK__DEFAULT_AUTHENTICATION_CLASSES = [‘rest_framework.authentication.SessionAuthentication’, ‘pulpcore.app.authentication.PulpRemoteUserAuthentication’]”

We are able to query the /api/v3/status/ on the new 3.22.21 server but having issues with other queries even we bypass the webserver authentication on the webserver. The only difference we can think of is the python modules on the new and old servers are not exact the same. We tried to downgrade pulpcore and django to match the old server and it didn’t help. Does both running 3.22.4 and 3.22.21 at the same time may cause an issue? Does pulp have a db schema change from 3.22.4 to 3.22.21. Any suggestions would be greatly appreciated. We really don’t know what to try at this time.

ggainey · February 2, 2024, 7:21pm

Yeah, if you’re using chrony you should be fine.

I don’t see anything the pulpcore changelog that suggests it would affect this.

When you say “the python modules on the new and old servers are not exact the same” - how not-exact are they?

I am really puzzled about what could be causing this problem, alas

ipanova · February 5, 2024, 5:07pm

It can be related to the encripted fields Migration of encrypted fields between 3.14 and 3.16 is broken for some remotes · Issue #2327 · pulp/pulpcore · GitHub
If I recall correctly we’ve been fixing this multiple times, I just could not find other PRs/Issues to confirm where landed that ‘last’ fix.

bli111 · February 6, 2024, 10:50pm

Here is the python modules version $ sdiff /tmp/old /tmp/new
aiodns 3.0.0 aiofiles 22.1.0 aiohttp 3.8.3 aiohttp-xmlrpc 1.5.0 aioredis 2.0.1 aiosignal 1.3.1 ansible 7.3.0 ansible-builder 1.2.0 ansible-compat 3.0.1 ansible-core 2.14.3 ansible-lint 6.8.6 asgiref 3.6.0 asyncio-throttle 1.0.2 async-lru 1.0.3 async-timeout 4.0.2 attrs 21.4.0 backoff 2.1.2 bindep 2.11.0 black 22.10.0 bleach 3.3.1 bleach-allowlist 1.0.3 bloomberg.guts.api 6.4.0 bloomberg.guts.events 0.3.1 boto3 1.26.15 botocore 1.29.94 bracex 2.3.post1 build 0.10.0 certifi 2022.12.7 cffi 1.15.1 chardet 5.1.0 charset-normalizer 2.1.1 click 8.1.3 contextlib2 21.6.0 createrepo-c 0.20.1 cryptography 38.0.4 defusedxml 0.7.1 diff-match-patch 20200713 distro 1.8.0 Django 3.2.18 django-currentuser 0.5.3 django-filter 22.1 django-guid 3.3.0 django-import-export 3.0.2 django-lifecycle 1.0.0 django-readonly-field 1.1.2 djangorestframework 3.13.1 djangorestframework-queryfields 1.0.0 django-storages 1.13.2 drf-access-policy 1.1.2 drf-nested-routers 0.93.4 drf-spectacular 0.25.0 dynaconf 3.1.11 et-xmlfile 1.1.0 filelock 3.10.0 flake8 3.9.2 frozenlist 1.3.3 galaxy-importer 0.4.6 gitdb 4.0.10 GitPython 3.1.31 gunicorn 20.1.0 httpie 3.2.1 idna 3.4 importlib-metadata 6.1.0 > importlib-resources inflection 0.5.1 iniparse 0.5 Jinja2 3.1.2 jmespath 1.0.1 jsonschema 4.17.3 libcomps 0.1.15.post1 lxml 4.9.2 Markdown 3.4.1 markdown-it-py 2.2.0 > markdown-it-py MarkupPy 1.14 MarkupSafe 2.1.2 mccabe 0.6.1 mdurl 0.1.2 modulemd 1.3.3 multidict 6.0.4 mypy-extensions 1.0.0 naya 1.1.1 nose 1.3.7 odfpy 1.4.1 openpyxl 3.1.2 Package Version packaging 23.0 Parsley 1.3 pathspec 0.11.1 pbr 5.11.1 pip 23.1.2 pip-tools 6.13.0 platformdirs 3.1.1 productmd 1.33 protobuf 4.21.12 psycopg2 2.9.5 psycopg2-binary 2.9.5 pulp-ansible 0.16.1 pulp-cli 0.18.0 pulpcore 3.22.4 pulpcore-client 3.21.3 pulp-deb 2.20.2 pulp-file 1.12.0 pulp-glue 0.18.0 pulp-rpm 3.19.3 pulp-rpm-client 3.18.9 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycairo 1.23.0 pycares 4.3.0 pycodestyle 2.7.0 pycparser 2.21 pyflakes 2.3.1 Pygments 2.14.0 PyGObject 3.42.2 pygtrie 2.5.0 pyparsing 3.0.9 pyproject_hooks 1.0.0 pyrsistent 0.19.3 PySocks 1.7.1 python-dateutil 2.8.2 python-debian 0.1.49 python-gnupg 0.5.0 python-ldap 3.4.3 python-magic 0.4.27 pytz 2022.7.1 PyYAML 5.4.1 redis 4.4.0 requests 2.28.2 requests-toolbelt 0.10.1 > requests-toolbelt requirements-parser 0.5.0 resolvelib 0.8.1 rhsm 1.19.2 rich 13.3.2 ruamel.yaml 0.17.21 ruamel.yaml.clib 0.2.7 s3cmd 2.3.0 s3transfer 0.6.0 > s3transfer schema 0.7.5 scikit-build 0.16.7 semantic-version 2.10.0 setuptools 50.3.2 six 1.16.0 smmap 5.0.0 solv 0.7.22.post2 sqlparse 0.4.3 subprocess-tee 0.4.1 tablib 3.3.0 toml 0.10.2 tomli 2.0.1 types-setuptools 67.6.0.5 typing_extensions 4.5.0 uritemplate 4.1.1 urllib3 1.26.15 urlman 2.0.1 url-normalize 1.4.3 wcmatch 8.4.1 webencodings 0.5.1 wheel 0.40.0 whitenoise 6.2.0 xlrd 2.0.1 xlwt 1.3.0 yamllint 1.29.0 yarl 1.8.2 zipp 3.15.0 comparison between old and new server. Could any of the newer version of the module cause this issue?
aiodns 3.0.0
aiofiles 22.1.0
aiohttp 3.8.3
aiohttp-xmlrpc 1.5.0
aioredis 2.0.1
aiosignal 1.3.1
| ansible 8.7.0
| ansible-builder 3.0.0
| ansible-compat 4.1.11
| ansible-core 2.15.8
ansible-lint 6.8.6
| asgiref 3.7.2
asyncio-throttle 1.0.2
async-lru 1.0.3
| async-timeout 4.0.3
| attrs 22.2.0
| backoff 2.2.1
bindep 2.11.0
black 22.10.0
bleach 3.3.1
bleach-allowlist 1.0.3
| bloomberg.guts.api 6.6.1
| bloomberg.guts.events 0.3.3
boto3 1.26.15
| botocore 1.29.165
| bracex 2.4
| build 1.0.3
| certifi 2023.11.17
| cffi 1.16.0
| chardet 5.2.0
charset-normalizer 2.1.1
click 8.1.3
contextlib2 21.6.0
| createrepo_c 0.20.1
cryptography 38.0.4
defusedxml 0.7.1
| diff-match-patch 20230430
| distro 1.9.0
| Django 3.2.23
django-currentuser 0.5.3
django-filter 22.1
django-guid 3.3.0
django-import-export 3.0.2
django-lifecycle 1.0.0
django-readonly-field 1.1.2
| djangorestframework 3.14.0
djangorestframework-queryfields 1.0.0
| django-storages 1.14.2
| drf-access-policy 1.3.0
drf-nested-routers 0.93.4
drf-spectacular 0.25.0
dynaconf 3.1.11
et-xmlfile 1.1.0
| filelock 3.13.1
| flake8 6.1.0
| frozenlist 1.4.1
| galaxy_importer 0.4.19
| gitdb 4.0.11
| GitPython 3.1.41
gunicorn 20.1.0
| httpie 3.2.2
| idna 3.6
| importlib-metadata 7.0.1
5.0.7
inflection 0.5.1
iniparse 0.5
Jinja2 3.1.2
jmespath 1.0.1
jsonschema 4.17.3
| jsonschema-specifications 2023.12.1
| libcomps 0.1.20.post1
| lxml 5.1.0
| Markdown 3.5.2
3.0.0
MarkupPy 1.14
| MarkupSafe 2.1.4
| mccabe 0.7.0
mdurl 0.1.2
modulemd 1.3.3
multidict 6.0.4
mypy-extensions 1.0.0
naya 1.1.1
nose 1.3.7
odfpy 1.4.1
openpyxl 3.1.2
Package Version
| packaging 23.2
Parsley 1.3
| pathspec 0.12.1
| pbr 6.0.0
| pip 23.3.2
pip-tools 6.13.0
| platformdirs 4.1.0
productmd 1.33
protobuf 4.21.12
psycopg2 2.9.5
| psycopg2-binary 2.9.9
pulp-ansible 0.16.1
| pulp-cli 0.23.1
| pulpcore 3.22.21
<
pulp-deb 2.20.2
pulp-file 1.12.0
| pulp-glue 0.23.1
pulp-rpm 3.19.3
| pyasn1 0.5.1
| pyasn1-modules 0.3.0
| pycairo 1.25.1
| pycares 4.4.0
| pycodestyle 2.11.1
<
pycparser 2.21
| pyflakes 3.1.0
| Pygments 2.17.2
| PyGObject 3.44.1
pygtrie 2.5.0
| pyparsing 3.1.1
pyproject_hooks 1.0.0
| pyrsistent 0.20.0
PySocks 1.7.1
python-dateutil 2.8.2
python-debian 0.1.49
python-gnupg 0.5.0
| python-ldap 3.4.4
python-magic 0.4.27
| pytz 2023.3.post1
| PyYAML 6.0
redis 4.4.0
| referencing 0.32.1
| requests 2.31.0
1.0.0
requirements-parser 0.5.0
| resolvelib 1.0.1
rhsm 1.19.2
| rich 13.7.0
| rpds-py 0.17.1
| ruamel.yaml 0.17.40
| ruamel.yaml.clib 0.2.8
| s3cmd 2.4.0
0.6.2
schema 0.7.5
| scikit-build 0.17.6
semantic-version 2.10.0
setuptools 50.3.2
six 1.16.0
| smmap 5.0.1
solv 0.7.22.post2
| sqlparse 0.4.4
subprocess-tee 0.4.1
| tablib 3.5.0
toml 0.10.2
tomli 2.0.1
| types-setuptools 69.0.0.20240115
| typing_extensions 4.9.0
uritemplate 4.1.1
urllib3 1.26.15
urlman 2.0.1
url-normalize 1.4.3
| wcmatch 8.5
webencodings 0.5.1
| wheel 0.42.0
whitenoise 6.2.0
xlrd 2.0.1
xlwt 1.3.0
| yamllint 1.33.0
yarl 1.8.2
| zipp 3.17.0

bli111 · February 6, 2024, 11:01pm

I also noticed the similar issue in the past. After migration, we got 500 error to query the remotes. The most recent case happened we tried to migrate 3.22.4 to 3.44.1. We had no idea how to troubleshot this. We reinitialize the database and start from scratch. Is there a generic solution to fix this issue or the fixes depend on the migration path?

ipanova · February 7, 2024, 11:53am

have you rotated any encryption keys?

bli111 · February 7, 2024, 4:10pm

We use the same settings.py which has the secret_key. Do you mean the DB_ENCRYPTION_KEY? I don’t have it in the settings. The pulp_installer generated DB_ENCRYPTION_KEY automatically. I just checked the content of /etc/pulp/certs/database_fields.symmetric.key. The content is the same on all 8 old 3.22.4 servers. I see two different keys on the newly installed 3.22.21 servers. I wonder why they are different since we use the same pulp_installer. Should we explicitly set in the setting.py?

ipanova · February 7, 2024, 4:48pm

You won’t like what i will tell you - but we no longer support pulp-installer since already 1year+.
So please switch to containerized deployment and we can discuss issues you encounter.

And yes, you receive this error because your db encryption keys are different. Why it happened - I don’t know off-hand. In later versions of pulpcore we added support to accept list of keys, but you won’t be able to upgrade to it because last supported pulpcore version with installer was 3.22.

bli111 · February 7, 2024, 8:50pm

Unfortunately, it will take much longer time to use containerized deployment to upgrade to latest version. We are looking for a quick way to expand current production capacity. I noticed the the latest 3.22 was released about a week ago. I am wondering if we can simply copy the /etc/pulp/certs from the old servers to the new servers after running the pulp-installer. Will this approach work?

ipanova · February 8, 2024, 12:15pm

If you used to have same key on all servers and now just 2 of them have different key, then just make sure you have same key everywhere.