RPM Sequoia: A Sequoia-based backend for the RPM Package Manager

By Neal | April 27, 2023

Fedora 38 is out, and unsurprisingly it comes with a lot of shiny, new things. One especially interesting novelty for readers of this blog is that this is the first release of Fedora in which the RPM Package Manager uses Sequoia to verify packages. This blog post is the story of how that came to be.

OpenPGP in RPM

A package manager’s main responsibility is to install packages. This includes checking that a package’s dependencies are satisfied, unpacking it, and doing any necessary configuration. But, before a package manager does any of those things, it first needs to authenticate the package. Failing to do this is an invitation for a complete system compromise. Like many other package managers, RPM does this using digital signatures.

Checking that a package’s digital signature is mathematically correct is necessary, but not sufficient to authenticate a package. It is also necessary to check that the signature came from a person or organization the user is willing to rely on. This is the role of a public key infrastructure (PKI). As is popular in the FLOSS ecosystem, RPM uses OpenPGP for its PKI.

The RPM developers added OpenPGP support to RPM in 1996, just a few months after the first commit. That version called out to the pgp command-line tool. Support for using gpg was added two years later, shortly after its initial release at the end of 1997. In October 2001, the RPM developers removed shelling out to pgp or gpg in favor of an internal OpenPGP implementation with the comment: its “at least as good as pgp/gpg on verify.”

Over the past two decades, the RPM developers have maintained RPM’s OpenPGP implementation, but according to Panu Matilainen, RPM’s current maintainer, its development was primarily driven by necessity:

[The signature verification]’s not the most loved subsystem of rpm, exactly. Nobody ever stepped up to do the major rework (I would say redesign but that would imply a previous design…) it needs, so whatever “interface” there is, is pretty much all ad-hoc added for whatever the current need was.

In his recent call to replace RPM’s OpenPGP implementation, Panu elaborates that this situation arose because the people who work on RPM are mostly interested in packaging, not cryptographic minutia. And, to avoid this distraction, RPM should outsource this responsibility:

There’s something seriously wrong when a significant percent of package manager development discussions is about the OpenPGP specification and its interpretation in the RPM context. This is negatively impacting development of RPM “core business”, to the point that this has to stop. There’s exactly one way to stop it, and that’s getting rid of the internal parser, one way or the other.

Later, he wrote about the switch to Sequoia:

I doubt many people realize just how thin the ice is (and has always been) with the existing parser. I consider this step a matter of survival, and ultimately some legacy content becoming harder to use is an acceptable tradeoff for that. The main, direct benefit to Fedora is improved security and standards-compliance (RFC-4880) in one of the corner-stones of the whole distribution. Longer term, we can expect better error messages and other functional improvements regarding key and signature handling.

In short, RPM’s internal OpenPGP parser never got the attention that a security-relevant component needs. And as a result, the API is awkward, and the implementation is barely sufficient.

Towards a new OpenPGP Backend

Justus Winter, a co-founder of the Sequoia project, spent some time studying RPM’s OpenPGP implementation. In October 2021, he sent a mail to the rpm-maint mailing list summarizing his findings. One of his main concerns was about the implementation’s correctness. In particular, he was worried about how RPM interprets certificates, which is complicated. He also anticipated Panu’s point that the RPM developers are primarily interested in using OpenPGP, not implementing it. He suggested:

[M]y strategy would be to decouple the current implementation by clearly defining the public API, then provide a drop-in replacement for that API that can be enabled at compile-time.

A lively discussion followed. One of the main concerns, which Justus also raised in his initial mail, was that using Sequoia would make Rust a dependency, and that would seriously complicate bootstrapping a distribution. To avoid that, the internal implementation would have to be maintained for the foreseeable future. Panu concluded:

I’m not at all eager to gain a Rust dependency and there’ll be somewhat more (not less) code to maintain, but as per the plan above I think this sounds like a net positive for us.

Having obtained consent from RPM’s primary developer, we began the implementation work.

Porting RPM to Sequoia

I took on the task of adding Sequoia to RPM. It quickly became clear that it would be better to split Justus’ plan into two separate steps.

Justus proposed designing and implementing a new API. Unfortunately, RPM’s OpenPGP implementation isn’t an implementation detail; librpmio exposes the OpenPGP functionality as part of its public API. Thus, moving to a new API would either mean breaking RPM’s API, or implementing both the existing API, and the new API. And because RPM should be ported to the new API, and the internal implementation should continue to work, a new API would have to be implemented twice: once with Sequoia, and once with the internal OpenPGP implementation. After spending a bit of time with the internal implementation, I discovered it quite challenging to extend.

My revised plan was to first implement the existing API using Sequoia, and get that change into RPM. Then I wouldn’t have to touch the internal implementation too much. After that was done, and everyone was happy, we would design and implement a new API. And, if we were lucky, the internal implementation could also then be removed.

Because RPM is written in C and Sequoia is written in Rust, any integration needs to use a foreign function interface (FFI). Rust makes it straightforward to write functions that can be called from C, so by itself, this isn’t a problem. But not every Rust function can be called from C. For instance, unlike C, Rust supports generic types. For pretty much any non-trivial API, this means manually wrapping and unwrapping objects as they cross the language boundary. This wrapping and unwrapping can be error prone. Further, C and Rust have different fundamental types. In Rust, a string is a tuple consisting of a length and a pointer, and the content is guaranteed to be valid UTF-8 data. In C, a string is a zero-terminated array of bytes. This impedance mismatch is another potential source of bugs.

There are two ways to do this type of integration. Sequoia could provide a thin, generic wrapper, which exposes most of the underlying Rust library’s functionality, and which follows the target language’s coding conventions. We tried this with sequoia-ffi. Because sequoia-openpgp is a low-level library, it meant wrapping a huge number of functions. That is time consuming and repetitive work, which we found hard to automate, and easy to mess up for the aforementioned reasons. The alternative is to implement a point solution. That is, we implement an application-specific library in Rust, which exposes exactly the high-level interface that the application requires. We’ve done this several times, e.g., in cooperation with p≡p and anonaddy, and it worked very well. The shims are small and easy to understand, and the API is just a dozen or so functions, which deal with high-level objects.

Based on our negative experience implementing a generic wrapper, and our multiple positive experiences implementing point solutions, I decided to try the latter. The result is rpm-sequoia, which is now part of the RPM project.

I submitted the initial pull request at the end of March 2022. This version passed most of the test suite, but I needed some feedback. I had a few questions about how to best proceed, and, more importantly, whether I should bother proceeding at all.

Panu reacted positively to the pull request:

Oh and to make it absolutely clear, this effort is very much appreciated.

Over the next few weeks, there was a fair amount of back and forth, as I implemented the remaining bits, and adjusted the implementation so everyone was happy. I also took the opportunity to improve error reporting when importing certificates, which was a problem others had reported, and one that I encountered when debugging the differences in behavior between the internal OpenPGP implementation, which is rather liberal, and Sequoia, which is more strict. A month later, at the end of April, Panu merged the changes. 🥳.

Shortly after, Fabio Valentini, aka decathorpe, who maintains all of the Sequoia-related packages in Fedora, packaged rpm-sequoia.

RPM 4.18

The next release of RPM, version 4.18, was planned for fall 2022, several months later. Before then, a few issues came up. In particular, there was a small misunderstanding.

Around the time the Sequoia-related code was merged, Panu removed several OpenPGP-related functions, which I assumed rpm-sequoia didn’t need to implement. It turned out that the so-name bump was only happening in 4.19, so implementations of those would have to be added if the Sequoia-backend should be part of the 4.18 release. Implementing deprecated functions isn’t terribly rewarding, but I decided it was better than delaying the integration for a year.

With that merged, the Sequoia backend was added to 4.18.0-rc1.

Into Fedora

With the release of 4.18.0-rc1 in September 2022, I sent a heads up to the Fedora developer community, and asked for input on the transition to the new backend. We got some helpful feedback, and were able to address some concerns.

First, Fedora and RHEL have largely standardized on the OpenSSL cryptographic library to reduce the amount of cryptographic code in the base system. Using RPM with Sequoia would have added a dependency on Nettle. Using fewer cryptographic libraries is helpful for minimizing the maintenance burden, FIPS certification, and reducing the size of containers.

Happily, Panu had previously made us aware of this concern, and Wiktor Kwapisiewicz, a long-time Sequoia developer, had already begun work on an OpenSSL backend. A few weeks after that mail, Wiktor opened a merge request to add OpenSSL support to sequoia-openpgp. And, at the beginning of 2023 we released version 1.13.0 of sequoia-openpgp with support for using OpenSSL as the cryptographic library.

Another concern had to do with the system’s cryptographic policy. Fedora uses a system-wide cryptographic configuration system. This allows a system administrator to disable an algorithm, say, SHA-1, for one application, or for all applications using the same tool.

I was only vaguely aware of the crypto policy, but adding support for it required no special changes to sequoia-openpgp. In sequoia-openpgp, all operations that make use of cryptographic primitives take a policy object, which controls what primitives are acceptable. What needed to be done was to read the policy from a configuration file. As sequoia-openpgp is a library, and applications may have different requirements, I decided to write a separate library, sequoia-policy-config, to parse a common configuration format, and configure a policy object appropriately. In the end, this turned out to be a good decision, as I’ll return to below.

The final concern was again about Rust’s portability. Fabio pointed out that it is actually quite good for Fedora’s target architectures, and no further objections were raised.

Shortly after this thread, Panu wrote a Fedora change proposal for switching RPM to Sequoia in Fedora 38 (tracking issue):

Rpm has been using it’s own simple and flawed OpenPGP parser ever since v4.0 or so. There’s now a much more advanced alternative in rpm-sequoia, we should switch rpm to use it instead.

Initial plan for this change is early in Fedora 38 release process to have time to deal with any potential teething issues.

The Fedora Steering Committee (FESCo) voted on the proposal and unanimously accepted it.

Teething Issues, Part I

Shortly after FESCo’s go ahead, the first version of RPM using Sequoia was picked up by COPR. COPR is one of Fedora’s two build systems (the other being Koji). It provides a testing ground for packages that aren’t ready for inclusion in Fedora, and a way to build and distribute packages that aren’t intended for inclusion in Fedora.

This was the first big test of using the Sequoia backend in practice. To put it succinctly: it failed fast.

Based on a few more random samples, it would appear that everything signed by COPR is broken this way now.

A bit of investigation revealed the problem: COPR was signing packages using v3 signatures, which, although not completely deprecated by the last revision of the OpenPGP standard, have been largely deprecated in practice, and weren’t supported by Sequoia. Unfortunately, RPM’s test suite didn’t pick this up, as the test data only includes v4 signatures!

Panu did some investigation and found that a long time ago, obs-sign was configured to create v3 signatures and, like many workarounds, forgotten:

So, basically everything signed by obs-signd is affected as it defaults to OpenPGP v3 signatures. And that being used by OBS and multiple other places for signing rpms, this affects at least

  • opensuse (and so presumably their enterprise offerings too but can’t verify that)
  • copr
  • rpmfusion

I don’t know what RHEL is signed with, but packages in RHEL 7-9 (didn’t bother with older) are signed using OpenPGP V3 signatures too.

So there really is only one conclusion to make: this is a no-go until Sequoia adds support for verifying V3 signatures. Or the world catches up, which is going to be years before all relevant content signed with V3 has gone dropped out of relevance, even if everybody started just now.

(Interestingly, Koji, Fedora’s main build system, was creating v4 signatures and not v3 signatures. But, this was just due to good luck. Whereas COPR uses obs-sign to sign packages, Koji uses Sigul. In 2020, the developers of Sigul switched from using GnuPG v1 to GnuPG v2 as part of adding support for the Linux Integrity Measurement Architecture (IMA) signature scheme. Unlike GnuPG v1, GnuPG v2 refuses to generate v3 signatures even when passed --force-v3-sigs, which Sigul still passes to gpg, and which GnuPG v2 silently ignores. So Fedora fortuitously updated to v4 signatures around 2020, and nobody noticed a thing!)

Happily, although v3 is not the preferred way to create OpenPGP signatures, it is not cryptographically broken. I added support for parsing v3 signatures to sequoia-openpgp, and to rpm-sequoia. A few days later, we were back in the game.

Panu shared the following post-mortum analysis:

Status update across the board:

A couple of interesting bits found during investigating all this:

GnuPG only started creating v4 signatures by default as late as 1.4.8 released in December 2007, and for example RHEL 5 released earlier that year carried 1.4.5 through its lifetime all the way up to 2017 (and 2020 with extended life support). I didn’t check, but I’d assume other enterprise distros to be in similar position here. Signatures created on RHEL 6 and newer are v4 by default (both gpg --sign and rpmsign), so that’s pretty old news too.

Fedora and EPEL appear to have switched to v4 signatures during 2020, but I haven’t found any explicit mention of this. Which makes me suspect this was accidental from Sigul switching to use gnupg2 internally at that time, gnupg2 which has ignored --force-v3-sigs even if passed to it since 2014.

All in all, I’d consider it a success. Yes, there was an issue. But we fixed it, and we discovered some latent infrastructure problems, which have now been addressed.

Teething Issues, Part II

Over the next few months all was fairly quiet. I privately asked Panu what to expect, and he replied:

Barring any new major surprises, this is the part where we just sit back and enjoy the ride. Rawhide will continue its perpetual churn, but Fedora 38 will get branched from it during early February and then released towards end of April, assuming the schedule holds.

Rawhide gets a surprising amount of testing for what it is, but the wider exposure starts around beta release. So that, and the first couple of months after final release are times where second and third waves of bug reports on new stuff typically happen, if they happen.

Panu was right. Things got exciting as we approached the finish line. In February, two months prior to the planned Fedora 38 release, Kamil Páral brought two major issues to our attention: RPMs that are installed (like Google Chrome) but considered insecure prevent system updates, and can’t be removed, and, relatedly, lots of third-party repositories use out-dated cryptography. Ouch! We came up with three fixes to address these problems.

First, Panu explained that the packages using weak cryptography couldn’t be uninstalled or updated, because RPM checks the signatures of installed packages to provide protection against tampering. To address this, we changed the signature verification functionality to return not just success or failure, but to also distinguish a valid signature that relies on legacy cryptography. This not only nicely solves the problem with checking installed packages for validity, but it improves the user experience, because users better understand why a package can’t be installed.

Second, because it was much too short of a time until the release of Fedora 38, FESCo voted to reenable the use of SHA-1 and 1024-bit DSA keys for Fedora 38:

FESCo agrees to block Beta for this issue. In order to unblock, RPM must accept SHA-1 hashes and DSA keys for Fedora 38, ideally with a deprecation warning that it will be disabled in F39. FESCo strongly advises against allowing these algorithms elsewhere, but will accept that for F38 if it’s the only achievable, timely solution.

Most people are not terribly happy about this. But, the result is still an improvement relative to the status quo, and because we’ve become aware of the problem, people are working on fixing the underlying issue. In particular, people are reaching out to the maintainers of third-party repositories to make them aware of the problem. (Note: if you need to update your certificate, because it uses SHA-1, sq-keyring-linter --fix might help.)

To accommodate this change, I modified rpm-sequoia to use an application-specific policy file rather than the general Sequoia-specific policy file. In this way, other applications that use Sequoia, and the generic Sequoia policy file won’t accept weak cryptography by default.

The last change was about improving usability. There are a bunch of reasons a signature may be considered invalid: the certificate that made the signature may violate the system’s policy, the signature itself may violate the system’s policy, an artifact may be malformed, etc. And, given that Sequoia is more strict than RPM’s internal OpenPGP implementation, these errors will occur more often, because some packages that were installable in the past will no longer be installable in Fedora 38. This will be further exacerbated as the cryptographic policy is tightened in the future.

To deal with this in the short term, we decided to introduce parallel versions of a couple of RPM’s functions. These functions behave identically to their existing counterparts, but they also return a string, which contains any errors or lints.

Like many Rust programs, rpm-sequoia doesn’t just work with error codes, but it builds up error chains where each link includes a description of what was going on. This is extremely helpful for understanding both the high-level reason that something resulted in an error, as well as any underlying reasons.

For these new functions, rpm-sequoia also doesn’t just return the first error that it encounters, but, like many linters, tries to collect all of the reasons that an operation is not possible. These are returned in the lint argument.

Although I’m biased, I don’t think it is a stretch to say that the quality-of-life improvements are dramatic. Whereas before RPM would just give a generic “signature is bad” error, it now displays detailed information about why the signature is bad. Consider installing Google Chrome with a cryptographic policy that rejects weak cryptography:

$ update-crypto-policies --set FUTURE
$ rpm -i google-chrome-stable-109.0.5414.119-1.x86_64.rpm
error: Verifying a signature using certificate
      4CCA1EAF950CEE4AB83976DCA040830F7FAC5991 (Google, Inc. Linux Package
      Signing Key linux-packages-keymaster@google.com):
  1. Signature 02b3 created at Mon Jan 23 21:23:32 2023 invalid: signature
     relies on legacy cryptography
      because: Policy rejected non-revocation signature (Binary) requiring
        collision resistance
      because: SHA1 is not considered secure since 1970-01-01T00:00:00Z
  2. Certificate A040830F7FAC5991 invalid: policy violation
      because: No binding signature at time 2023-01-23T21:23:32Z
      because: Policy rejected non-revocation signature (PositiveCertification)
        requiring second pre-image resistance
      because: SHA1 is not considered secure since 1970-01-01T00:00:00Z
warning: google-chrome-stable-109.0.5414.119-1.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 7fac5991: NOTTRUSTED

The above output shows that both the signature and the certificate are problematic. Similarly, signatures made with go’s OpenPGP library are not compliant and are rejected by Sequoia. The output says not only what is wrong, but provides maintainers actionable advice:

$ rpm -i intel-oneapi-common-licensing-2023.1.0-2023.1.0-43473.noarch.rpm
error: intel-oneapi-common-licensing-2023.1.0-2023.1.0-43473.noarch.rpm: Header RSA signature: BAD (package tag 268: invalid OpenPGP signature: Parsing an OpenPGP packet:
  Failed to parse Signature Packet
      because: Signature appears to be created by a non-conformant OpenPGP
        implementation, see https://github.com/rpm-software-management/rpm/issues/2351.
      because: Malformed MPI: leading bit is not set: expected bit 8 to be
        set in      101 (5))
error: intel-oneapi-common-licensing-2023.1.0-2023.1.0-43473.noarch.rpm cannot be
  installed

This support has already been added to rpm-sequoia, and to rpm. Unfortunately, the changes were too late for the official Fedora 38 release, but an updated version of RPM with these improvements will be pushed to Fedora 38 in the coming days.

The Fedora 38 Release

On April 18, 2023, Fedora 38 was released. I have to admit that I was still a bit nervous, and the few days after the release I kept searching the internet for “rpm sequoia” looking for complaints that we broke something. Happily, I didn’t find anything serious!

Sequoia is now the default OpenPGP implementation for RPM. The internal implementation will stick around for the foreseeable future to facilitate bootstrapping, and for distributions like openSUSE that prefer it.

But this is hardly the end of the story. We still want to redesign RPM’s API. DNF5 uses both GPGME and librpmio, and over the past few months, we’ve been collaborating with the developers to add missing functionality to librpmio to reduce the number of dependencies that they have. And some other projects, like OSTree are also considering moving to Sequoia.

Financial Support

Since the start of the project over five years ago, the p≡p foundation financially supports the people who work on Sequoia. In 2021, the NLnet foundation awarded us six grants as part of the NGI Assure program. And in 2022, we received a grant from the Sovereign Tech Fund.

We are actively looking for additional financial support to diversify our funding.

You don’t need to directly use Sequoia to be positively impacted by it. We’re focused on creating tools for activists, lawyers, and journalists who can’t rely on centralized authentication solutions. So, consider donating. Of course, if your company is using Sequoia, consider sponsoring a developer (or two). Note: if you want to use Sequoia under a license other than the LGPLv2+, please contact the foundation.