Pulp-minimal:3.35 pulp-api -- metadata_signing_service HTTP/1.1 500 Internal Server Error

Janr · September 29, 2023, 12:54pm

Hi All

Thanks for the help.

Problem:

Up to pulp3.32 updating metadata_signing_service worked fine with an API call

Upgrading to pulp3.35 the API call to update a repository’s metadata_signing_service fails.

Expected outcome:

pulp-minimal:3.35 to work like 3.32

Pulpcore version:

  "component": "core",
  "version": "3.35.0",
  "package": "pulpcore",

pulp-minimal 3.35 930d5aab2167 (image ID)

Pulp plugins installed and their versions:

Operating system - distribution and version:

Images runs on SLES15SP5

Other relevant data:

Result on pulp3.35

https --auth admin:1234 PATCH https://ita101.group.net:30082/pulp/api/v3/repositories/rpm/rpm/018a6e8c-f3eb-79bc-a2af-0dbdb630dc19/ <<< {
“metadata_signing_service”: “/pulp/api/v3/signing-services/018a1806-035a-78f7-afd2-e374306a0a2d/”
}

HTTP/1.1 500 Internal Server Error
Access-Control-Expose-Headers: Correlation-ID
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 145
Content-Type: text/html; charset=utf-8
Correlation-Id: 249267770734485e9e65ddde102e3c06
Cross-Origin-Opener-Policy: same-origin
Date: Fri, 29 Sep 2023 12:39:09 GMT
Referrer-Policy: same-origin
Server: Caddy, gunicorn
X-Content-Type-Options: nosniff
X-Frame-Options: DENY


<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

Result on pulp3.32

https --auth admin:1234 PATCH https://ita101.group.net:30082/pulp/api/v3/repositories/rpm/rpm/018a6e8c-f3eb-79bc-a2af-0dbdb630dc19/  <<<  {
  "metadata_signing_service": "/pulp/api/v3/signing-services/018a1806-035a-78f7-afd2-e374306a0a2d/"
}

HTTP/1.1 202 Accepted
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 67
Content-Type: application/json
Correlation-Id: d76324441f4349c9b9ec9620c04b1fac
Cross-Origin-Opener-Policy: same-origin
Date: Fri, 29 Sep 2023 12:18:23 GMT
Referrer-Policy: same-origin
Server: Caddy, gunicorn
Vary: Accept
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "task": "/pulp/api/v3/tasks/018ae0e0-2ff1-7b47-ac5d-6da0cc6d9ee3/"
}

Container stdout log

2023-09-29T14:45:11.392908297+02:00 stdout F ('pulp [657fb55b76514495a63b77b1a572748b]: ::ffff:127.0.0.1 - admin [29/Sep/2023:14:45:11 +0200] "GET /pulp/api/v3/signing-services/?name=my_collection_signer_vault&offset=0&limit=1 HTTP/1.1" 200 3595 "-" "Pulp-CLI/0.20.3"',)
2023-09-29T14:45:16.191475823+02:00 stderr F pulp [fb84809e50194a7c8e6ff2df50d5a323]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/018a6e8c-f3eb-79bc-a2af-0dbdb630dc19/
2023-09-29T14:45:16.191475823+02:00 stderr F Traceback (most recent call last):
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 55, in inner
2023-09-29T14:45:16.191475823+02:00 stderr F     response = get_response(request)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 197, in _get_response
2023-09-29T14:45:16.191475823+02:00 stderr F     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
2023-09-29T14:45:16.191475823+02:00 stderr F     return view_func(*args, **kwargs)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
2023-09-29T14:45:16.191475823+02:00 stderr F     return self.dispatch(request, *args, **kwargs)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
2023-09-29T14:45:16.191475823+02:00 stderr F     response = self.handle_exception(exc)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
2023-09-29T14:45:16.191475823+02:00 stderr F     self.raise_uncaught_exception(exc)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
2023-09-29T14:45:16.191475823+02:00 stderr F     raise exc
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
2023-09-29T14:45:16.191475823+02:00 stderr F     response = handler(request, *args, **kwargs)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/app/viewsets/base.py", line 499, in partial_update
2023-09-29T14:45:16.191475823+02:00 stderr F     return self.update(request, *args, **kwargs)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/app/viewsets/base.py", line 484, in update
2023-09-29T14:45:16.191475823+02:00 stderr F     task = dispatch(
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/tasks.py", line 165, in dispatch
2023-09-29T14:45:16.191475823+02:00 stderr F     task = Task.objects.create(
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 87, in manager_method
2023-09-29T14:45:16.191475823+02:00 stderr F     return getattr(self.get_queryset(), name)(*args, **kwargs)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 658, in create
2023-09-29T14:45:16.191475823+02:00 stderr F     obj.save(force_insert=True, using=self.db)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/lib64/python3.8/contextlib.py", line 75, in inner
2023-09-29T14:45:16.191475823+02:00 stderr F     return func(*args, **kwds)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django_lifecycle/mixins.py", line 172, in save
2023-09-29T14:45:16.191475823+02:00 stderr F     save(*args, **kwargs)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 814, in save
2023-09-29T14:45:16.191475823+02:00 stderr F     self.save_base(
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 877, in save_base
2023-09-29T14:45:16.191475823+02:00 stderr F     updated = self._save_table(
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 1020, in _save_table
2023-09-29T14:45:16.191475823+02:00 stderr F     results = self._do_insert(
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 1061, in _do_insert
2023-09-29T14:45:16.191475823+02:00 stderr F     return manager._insert(
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 87, in manager_method
2023-09-29T14:45:16.191475823+02:00 stderr F     return getattr(self.get_queryset(), name)(*args, **kwargs)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1805, in _insert
2023-09-29T14:45:16.191475823+02:00 stderr F     return query.get_compiler(using=using).execute_sql(returning_fields)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1821, in execute_sql
2023-09-29T14:45:16.191475823+02:00 stderr F     for sql, params in self.as_sql():
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django_readonly_field/compiler.py", line 31, in as_sql
2023-09-29T14:45:16.191475823+02:00 stderr F     return super(ReadonlySQLCompilerMixin, self).as_sql()
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1745, in as_sql
2023-09-29T14:45:16.191475823+02:00 stderr F     value_rows = [
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1746, in <listcomp>
2023-09-29T14:45:16.191475823+02:00 stderr F     [
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1747, in <listcomp>
2023-09-29T14:45:16.191475823+02:00 stderr F     self.prepare_value(field, self.pre_save_val(field, obj))
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1686, in prepare_value
2023-09-29T14:45:16.191475823+02:00 stderr F     return field.get_db_prep_save(value, connection=self.connection)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/json.py", line 136, in get_db_prep_save
2023-09-29T14:45:16.191475823+02:00 stderr F     return self.get_db_prep_value(value, connection)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/json.py", line 103, in get_db_prep_value
2023-09-29T14:45:16.191475823+02:00 stderr F     value = self.get_prep_value(value)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 154, in get_prep_value
2023-09-29T14:45:16.191475823+02:00 stderr F     value = self.encrypt(value)
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 129, in encrypt
2023-09-29T14:45:16.191475823+02:00 stderr F     return [self.encrypt(v) for v in value]
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 129, in <listcomp>
2023-09-29T14:45:16.191475823+02:00 stderr F     return [self.encrypt(v) for v in value]
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 131, in encrypt
2023-09-29T14:45:16.191475823+02:00 stderr F     return force_str(_fernet().encrypt(force_bytes(json.dumps(value, cls=self.encoder))))
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 26, in _fernet
2023-09-29T14:45:16.191475823+02:00 stderr F     return MultiFernet([Fernet(key) for key in key_file.readlines()])
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 26, in <listcomp>
2023-09-29T14:45:16.191475823+02:00 stderr F     return MultiFernet([Fernet(key) for key in key_file.readlines()])
2023-09-29T14:45:16.191475823+02:00 stderr F   File "/usr/local/lib64/python3.8/site-packages/cryptography/fernet.py", line 39, in __init__
2023-09-29T14:45:16.191475823+02:00 stderr F     raise ValueError(
2023-09-29T14:45:16.191475823+02:00 stderr F ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
2023-09-29T14:45:16.193076486+02:00 stdout F ('pulp [fb84809e50194a7c8e6ff2df50d5a323]: ::ffff:127.0.0.1 - admin [29/Sep/2023:14:45:16 +0200] "PATCH /pulp/api/v3/repositories/rpm/rpm/018a6e8c-f3eb-79bc-a2af-0dbdb630dc19/ HTTP/1.1" 500 145 "-" "HTTPie/2.6.0"',)
2023-09-29T14:45:37.037306881+02:00 stdout F ('pulp [e005b0b0a0334911a12197967d03e3ca]: ::ffff:127.0.0.1 - - [29/Sep/2023:14:45:37 +0200] "HEAD /pulp/api/v3/ HTTP/1.1" 200 9546 "-" "Nomad/1.6.1 (+https://www.nomadproject.io/; go1.20.5)"',)

Thanks so much.
Jan

x9c4 · September 29, 2023, 4:32pm

This is complaining about the database field encryption key.
How do you run your containers?
Can you share the part of your settings where DB_ENCRyPTION_KEY is specified? Is there a volume mounted to /etc/pulp/certs containing a file database_fields.symmetric.key?

Janr · October 2, 2023, 7:51am

Hello x9c4

Thank you for pointing out DB_ENCRYPTION_KEY

I am using Hashicorp’s Nomad and Vault.

PULP_DB_ENCRYPTION_KEY=/secrets/database_fields.symmetric.key

Up to Pulp3.34 the thee lines were accepted in database_fields.symmetric.key

cat database_fields.symmetric.key ( ‘\n’ added to help explain)

 \n
      "N....removed.............="\n
 \n

Changing the Nomad template to not add:

head blank line
two spaces in front of the key
not add a trailing blank line,

fetching the encryption key made Pulp 3.35 work.

Nomad template stanza now. Note the dash (-) : -}} and {{-end

    template {
        data = <<-EOH
        {{ with secret "secret/data/nomad_psql" -}}
          {{.Data.data.dbf_symmetric_key | toJSON }}
        {{- end }}
        EOH

    destination = "${NOMAD_SECRETS_DIR}/database_fields.symmetric.key"
    }

Working implementation is:

cat database_fields.symmetric.key ( no newlines - no trailing spaces)

"N....removed...............="

ENV definition for container
PULP_DB_ENCRYPTION_KEY = "/secrets/database_fields.symmetric.key"

No mounting of the key in /etc/pulp/cert directory needed.

Pulp3.35 is now working.

Thank you for the help.
Much appreciated.

x9c4 · October 2, 2023, 12:22pm

Thank you. For detail explaining your solution!

I am still a little bit concerned now. When using the toJSON filter, your secret will be wrapped in ", right? I think you now have to accept the fact that those quotation-mark are part of your key. Not the worst thing in the world. XD (On closer inspection, the cryptography library seems to be resiliant to this quotation…)
You may want to take a look at our docs in case you want to rotate that key anyway:
https://docs.pulpproject.org/pulpcore/configuration/settings.html#db-encryption-key

Janr · October 3, 2023, 1:59pm

Thank you for the (") quotation warning.

XD (On closer inspection, the cryptography library seems to be resilient to this quotation…)

Seems this is true, the ‘new’ existing quote-less key worked.

Question please.

In an environment of an api/content/worker image running.
How would you stop the application on each to update database_fields.symmetric.key with
new/old key. (As mentioned in the URL above)

As the implementation is without any persistent storage at all, Save external PostgreSQL and the S3 artifact storage.
The only way I can do this, as I see it, is manually go in the container and update the key fie, do the ‘pulpcore-manager rotate-db-key’ and restart.

(Adding the old key before and/or after the new is possible by having the Nomad template adding both on a Nomad restart.)

Thank you.

x9c4 · October 4, 2023, 6:21am

You have the artifact storage, you template the db key from outside, and you have the settings stored somewhere, right? That should be sufficient to just restart the containers and even start new ones (please test first).

BTW:

Janr · October 4, 2023, 10:16am

Thank you once again and for the RFC

settings.py is created every time in a Nomad template, so are all else. ( fetched as 'artifact' )

We are finalizing the POC to take to production.

We need to log a RFE for the pulp-squeezer - Ansible.
Where would be the correct place for this please?

Thanks you.

x9c4 · October 4, 2023, 10:30am

That will naturally go here:

Heads up, requests in that subproject may take their time.

Janr · October 4, 2023, 1:50pm

Thank you.