Failed to sync redhat repos

Support
#1

*Problem:
syncing redhat repo failed

Expected outcome:

Pulpcore version:
3.22.4

Pulp plugins installed and their versions:
pulp-ansible 0.16.1
pulp-cli 0.19.0
pulp-deb 2.20.2
pulp-file 1.12.0
pulp-glue 0.19.0
pulp-rpm 3.19.3
pulp-rpm-client 3.19.3
pulpcore 3.22.4
pulpcore-client 3.22.4

Operating system - distribution and version:
rhel 8.6

Other relevant data:
We have been syncing from redhat for last 7 months. Recently we have been getting the below errors for most of redhat repos. Any idea why this happens and how to fix it?

    "traceback": "  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 460, in execute_task\n    result = func(*args, **kwargs)\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 398, in synchronize\n    remote = RpmRemote.objects.get(pk=remote_pk)\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/manager.py\", line 85, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py\", line 431, in get\n    num = len(clone)\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py\", line 262, in __len__\n    self._fetch_all()\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py\", line 1324, in _fetch_all\n    self._result_cache = list(self._iterable_class(self))\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/query.py\", line 68, in __iter__\n    for row in compiler.results_iter(results):\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1122, in apply_converters\n    value = converter(value, expression, connection)\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/pulpcore/app/models/fields.py\", line 104, in from_db_value\n    return force_str(self._fernet.decrypt(force_bytes(value)))\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/cryptography/fernet.py\", line 91, in decrypt\n    return self._decrypt_data(data, timestamp, time_info)\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/cryptography/fernet.py\", line 152, in _decrypt_data\n    self._verify_signature(data)\n  File \"/opt/utils/venv/pulp/3.9.7/lib64/python3.9/site-packages/cryptography/fernet.py\", line 136, in _verify_signature\n    raise InvalidToken\n"

I also notice rhel-7.6-server-eus-rpms didn’t have the issue when we use the same entitlement. We also test the same entitlement key on our dev instance with pulpcore 3.45 and it worked. It doesn’t look like this is an entitlement key issue.

#2

We’ve seen something like that when the encryption-key used by the pulp instance doesn’t match the one used to encrypt sensitive information (like the entitlement cert).

It may be that the encryption-key being used changed without calling rotate-db-key - see DB_ENCRYPTION_KEY for details.

#3

Thanks Grant. However, this version doesn’t have rotate-db-key. Not sure how we get to this stage. Is there a way to reset the key for this version 3.22.4?

Type 'pulpcore-manager help ’ for help on a specific subcommand.

Available subcommands:

[app]
add-signing-service
analyze-publication
datarepair-2327
datarepair-labels
download-log
dump-permissions
dump-publications-to-fs
handle-artifact-checksums
remove-plugin
remove-signing-service
reset-admin-password
rpm-datarepair
rpm-trim-changelogs
stage-profile-summary

[auth]
changepassword
createsuperuser

[contenttypes]
remove_stale_contenttypes

[django]
check
compilemessages
createcachetable
dbshell
diffsettings
dumpdata
flush
inspectdb
loaddata
makemessages
makemigrations
migrate
sendtestemail
shell
showmigrations
sqlflush
sqlmigrate
sqlsequencereset
squashmigrations
startapp
startproject
test
testserver

[django_extensions]
admin_generator
clean_pyc
clear_cache
compile_pyc
create_command
create_jobs
create_template_tags
delete_squashed_migrations
describe_form
drop_test_database
dumpscript
export_emails
find_template
generate_password
generate_secret_key
graph_models
list_model_info
list_signals
mail_debug
managestate
merge_model_instances
notes
pipchecker
print_settings
print_user_for_session
raise_test_exception
reset_db
reset_schema
runjob
runjobs
runprofileserver
runscript
runserver_plus
set_default_site
set_fake_emails
set_fake_passwords
shell_plus
show_template_tags
show_urls
sqlcreate
sqldiff
sqldsn
sync_s3
syncdata
unreferenced_files
update_permissions
validate_templates

[drf_spectacular]
spectacular

[rest_framework]
generateschema

[sessions]
clearsessions

[staticfiles]
collectstatic
findstatic
runserver

#4

Also I am able to see the ca_cert when query the remote. Does it mean the current DB_ENCRYPTION_KEY is still valid?

#5

If the DB_ENCRYPTION_KEY no longer matches the one used to store the Remotes in the first place, then…we’re in a not-great place. rotate- is useful, but not if the old-key has already been lost - so even if you had access to it it wouldn’t help.

The only things encrypted in remotes are client-key, and username/password and proxy-username/password - so you’ll always see ca-cert, that’s public and not a secret.

You could try creating a new remote with your client-cert/key, and a new repo that uses that remote, and try a sync. If that works - then it def feels like your DB_ENCRYPTION_KEY in settings isn’t what was originally used, and your remotes will prob need to have their client-cert/keys updated to replace the ones you can no longer decrypt.