Enforce package (deb and rpm) signatures

I’d like to propose (and help contribute) a new feature where Pulp is able to enforce a policy of THOU SHALT NOT UPLOAD UNSIGNED (dep and rpm) PACKAGES. I’m mostly interested in an installation-wide enforcement of that policy, however if it were configurable so that you could selectively enable it only for particular repos that’s fine too.

I started the discussion in pulp-dev and was told to bring it here, so I’ll just briefly summarize / link things that have already been talked about:

  • There exists today a SigningService, which allows you to call out to a shell script to sign packages. Maybe that could be extended to provide verification services too?
  • Would people want different signatures for eg beta repos? Maybe a default key and you allow per-repo overrides?
  • A way to make the verification script location more obvious / less confusing?
1 Like

Ok, I just saw the discussion on matrix, and I have a few thoughts to add:

  • In Debian world, it is pretty unusual to sign individual packages (though I believe it is possible), but almost unheard of not to sign your repository metadata. This provides integrity for the packages as well, since they are all referenced by checksum from the repository metadata. As a result, I am a little dubious how widespread a usecase this is for pulp_deb (of course I am not opposed to providing options).
  • Second: pulp_deb already provides some amount of repository metadata signature verification if the user provides a GPG key to a remote. The mechanism is currently pretty basic and could be improved in various ways, but it does already exist. (The way it works is that if a gpgkey is provided during sync, any Release files that cannot be verified via that gpgkey are simply discarded, which can cause nothing to be synced.)
  • I don’t think the analogy between signing service and some kind of proposed “verification service” makes sense. Signing services, with their messy script placement, only exist, because we did not want Pulp to take responsibility for handling users secret keys, and that is the interface we came up with. For verification, we don’t need secret keys, so we don’t need to jump through any hoops. Simply having a field on some model for users to provide keys to verify with is sufficient. What that verification actually entails will most likely be plugin specific. It will be for plugin authors to decide, so no external scripts needed.
1 Like

There’s an existing issue for the RPM plugin here, it hasn’t gotten much attention as of yet but it’s certainly a reasonable feature request and we’d be glad to work with you on implementing it.

I don’t know how much infrastructure could be shared between Deb and RPM. Probably not much, although the general approaches can be shared.

Some open questions:

  • Do we want to enforce that you can’t ever upload a raw artifact without a signature, or that you can’t create a Package content without a signature? The former would be much more challenging.
  • There is no good API currently for verifying RPM signatures, so probably you need to shell out to rpm --verify on the command line from within Pulp. But, I don’t immediately see any need for this to require a shell-script based API or anything like that.

In Debian world, it is pretty unusual to sign individual packages (though I believe it is possible), but almost unheard of not to sign your repository metadata. This provides integrity for the packages as well, since they are all referenced by checksum from the repository metadata.

In the RPM world it’s nearly the exact opposite, RPMs are typically signed, but the metadata itself is often not (although it is becoming more common over time). The idea being, wherever you get the package from you can be confident in its authenticity.

I don’t think the analogy between signing service and some kind of proposed “verification service” makes sense.

Agreed about this. There shouldn’t be any overlap between signing and verification.

You are taking the words right out of my mouth. I’m usually phrasing: Signing is hard. Verifying is easy. Let’s not make verifying hard for the sake of symmetry.
The plugin is really the place where the knowledge about a certain type of content and how a signature looks like converges. So I’d expect us to go the Pulp way, by looking at the pulp_deb implementation (verifying Release.gpg and InRelease), develop the feature in the corresponding plugins (ansible, container, deb and rpm should be in the loop here) and see if there are any valuable primitives to factor out into pulpcore.
Now that i’m thinking about this, i think we have some validation in pulp_ansible and pulp_container already.
What i really like is the idea of defining verification policies. We should try to make them consistent across plugins.

+1 to the idea of using the strategy of pioneering this in the plugins and if there are commonalities, later moving them to pulpcore. If we do this, it will allow @Stephen_Herr to basically work with the RPM and DEB teams to define this feature and implement it more quickly than consuming the code from pulpcore.

Big +1 to no external scripts. They don’t make sense for verification; they just aren’t needed. I expect it’ll basically be plugin-provided code that takes in a public key and verifies it based on the semantics of how the signature is provided.

1 Like

Something I learned after I submitted this is that Microsoft’s policy is that we should be upfront about working for Microsoft when interacting with open source communities, so let me state that right now: I work for Microsoft and this request stems from that fact. Microsoft has a very deep interest in solving the “supply chain security” problem.

So yes, we will be signing debs, even if the community doesn’t typically check for signatures on individual debs (debsig-verify is maintained by dpkg.org, so yes there is tooling for that). We will also be signing yum repodata, even if the yum world doesn’t typically check for that. We will sign all the things, and want to enforce signing all the things as a matter of policy.

I am unopinionated on this topic. In my use-case the packages are built and signed before people would be interested in pushing them to Pulp, so it seems easier to me to just reject the package upload immediately. But I understand that not everyone will have the same workflow.

This is my understanding too (thanks for the issue link). Which means that if the pulp_rpm plugin wants to validate signatures directly, the project will have a dependency on rpm being installed. Which maybe isn’t that big of a deal, rpm is actually available on deb-based systems too.

On the deb side debsig-verify is not available on rpm-based systems, however deb signing is actually a relatively simplistic thing, it just cats the content of the ar archive together and adds a detached signature. You can recreate the check that debsig-verify does with a 3-line bash script that requires only gpg and ar (and I think there are python libs that can replace ar, so you probably don’t even need that dependency) that looks something like this:

ar x *.deb
cat debian-binary control.* data.* > combined
gpg --verify _gpgorigin combined
1 Like

From the verification effort in pulp_container we have already moved from pulpcore.plugin.util import verify_signature to pulpcore.

It might also be possible to leverage https://github.com/Richterrettich/rpm-rs with some Python bindings, which I have some experience with doing. But let’s call that a backup plan for now, since the exact method of doing the verification is more of an implementation detail than the enforcement mechanisms Pulp would need to build around it.

Now that the broad approach has been mostly hammered out, I have a few more thoughts on implementation:

I gather from the above discussion, there need to be at least four possible values for something called verification_policy:

  • none: Do not require verification for anything (probably the default)
  • packages_only: Require signatures for packages but not for metadata.
  • metadata_only: Require signatures for metadata but not for packages
  • strict: Require signatures for both metadata and packages

Obviously not all of these make sense for all plugins (but they probably do for both deb and rpm). Some open questions:

  • Does that cover everything?
    • Is there perhaps a use case for differing verification policies depending on whether a given package is obtained via sync or via upload?
    • Where it is possible for things to be signed using multiple, or different GPG keys (within a single upstream repo), do users need to be able to specify that a specific key needs to have been used for a specific thing? (This could get really complicated fast…)
  • Where should the verification policy be set? Possible places might include “Pulp instance wide setting”, “Plugin wide setting”, on a repository, on a remote. (Remote does not capture package upload, so repository probably makes more sense?) A combination is also plausible, so perhaps users could set a plugin wide setting, and then override that plugin wide setting for individual repositories or remotes.

Whatever we choose here, I feel the interface for “verification plicies” between pulp_deb and pulp_rpm should try to be identical.

My other thought concerns the implementation on the pulp_deb side: The first thing to check, is if the python-debian library we are using already provides functions for verifying signed packages. If yes we can simply use that. If no, we might even consider opening a MR on the library, rather than simply adding that code to pulp_deb. (I have successfully contributed a small change to this library before. The only caveat is that the release cycle is a bit slow.) Even if we don’t end up going this route, we will almost certainly want to use the library to extract any constituent parts from the .deb package file.

Regarding the verification of pulp_deb repo metadata during sync, I think pulp_deb currently only allows a single GPG key to be specified for checking the signatures. However, it is perfectly legal for different distributions/releases within a single APT repo to be signed using different keys. I think it is also possible for a single distribution/release to be signed with multiple keys. This is just something to keep in mind while working on this feature.

@Stephen_Herr If you do start working on this, I am probably the person to ask any pulp_deb specific questions you may have. I would also encourage you to open a draft PR against pulp_deb early (even with a non-functional initial state), just to start that conversation.

1 Like