Using GPG Signing Service with Multiple pulp-worker Hosts

Support
#1

I’m running pulp-worker with Podman across multiple hosts. From what I understand, in order to use the signing service, both the GPG key and the signing service script need to be available on each worker instance.

I’ve read through the docs here:
Configure Signing Service

However, the guide doesn’t cover setups where workers are spread across different hosts.

My question is: Should the GPG key and signing script be baked into the pulp-minimal image to ensure all workers can access them? Or is there a recommended best practice for distributing keys and scripts across multiple worker hosts?

Thanks!

#2

I feel like the short answer is: “No, there is no recommended best practice for distributing keys and scripts across multiple worker hosts.”

I would say this is left deliberately vague. What Pulp really provides is an interface to an external service: “If you place a signing script/executable that is accessible from the workers, then Pulp will run some validation that this executable (in principle) provides the expected signing functionality and will then simply run it when something needs to be signed.”

What actually happens within this executable/signing script is deliberately left up to the Pulp admin. The reason is that every organization has different rules and requirements for how securely signing keys need to be kept. Within this context, the need to have this executable available on multiple distributed workers, as well as the lack of providing secrets and credentials via the Pulp interface is unfortunate. One could do something like build a signing script that calls on a single remote “signing service” (hence the name within Pulp) without the signing key being available on the Pulp workers at all.

The whole point is that Pulp is agnostic as to what you actually put in your signing script and where you store your keys. The only thing Pulp cares about is that calling the script with a file that is to be signed results in the relevant signature(s) being returned.

I would say that this interface is far from perfect. Anyone with ideas on how to improve it is welcome to open a ticket or community discussion. But be aware that “securing your signing keys” is a foundational hard security problem, so don’t expect any miracles!

1 Like
#3

There are probably a million more or less secure ways to do this. Yes, pulp only cares about being able to call a script (only from the pulpcore-worker process) that provides the signatures.
This script can then call to an online signing service, use a hardware crypto device, or “just” call into gpg.
In a containerized environment, I’d expect credentials / keys to be mounted as “secrets” into the container.

And yes, for the design, Pulp will never require to have access to the private keys so they can be properly protected. That’s why calling into a configured script as the most flexible interface was selected.

1 Like
#4

Thanks, that makes sense. I now understand that the signing script itself needs to be present on all worker images, since that’s what Pulp directly invokes. The handling of the keys, however, can be done in different ways depending on how the script is written — whether the key is local, provided at runtime, or delegated to a remote signing service.

This gives us flexibility to align with our own security requirements while still fitting into the Pulp interface.

Have you seen any examples of remote signing service setups in practice? It would be helpful to learn how others have approached this.

3 Likes
#5

I think you have the theory down. Personally I don’t know of any examples that turned the more complex scenarios into practice. Maybe someone else will chime in. If you are extra lucky you get to be the first! :wink:

1 Like