By Neal | June 6, 2019
For us, Sequoia is more than just a new OpenPGP implementation. Sequoia is firstly a project to rethink the OpenPGP ecosystem. We are convinced that OpenPGP has a lot of untapped potential. In particular, we think its flexible support for modeling trust (the web of trust is just one trust model built on top of OpenPGP’s certification mechanisms, which never had good tooling for advanced users never mind normal people), and the ability to update encryption & signature keys, and identity information without changing the key’s primary identifier are underappreciated and under used features.
To this end, we decided our first goal would be a low-level, policy-free library. And, after we nailed that, we’d work out a high-level, opinionated API, which covers common use cases. Our overarching goal is to make it easier to add support for OpenPGP to applications in a safe manner while allowing developers to override those high-level decisions that are inappropriate for their application in a straightforward fashion with minimal effort.
Currently, we are completing and polishing Sequoia’s low-level API and the corresponding C bindings. By the end of the summer, we hope that we’ll have gotten enough feedback that we can declare it stable, make a 1.0 release, and concentrate on the higher-level API and accompanying services.
The Low-Level Library
An important technical goal for Sequoia’s low-level library is to not simply implement the OpenPGP standard, but to expose a safe, easy-to-use, and policy-free interface. This means, for instance, that when an application signs a message with an OpenPGP key that has multiple signing-capable subkeys, our low-level library doesn’t just choose one of them, but makes it easy for the caller to select the right one for the task at hand.
A low-level library sounds like it would be hard to use. But our experience porting the p≡p cryptographic engine to Sequoia is that the PGP support requires less code using Sequoia, the official high-level interface to GnuPG, which the p≡p engine currently uses.
At this point, we feel like the low-level library implements about 99% of the OpenPGP standard. You can find details about what we’ve implemented, and what we haven’t as well as some caveats on our status page.
The last few percent of OpenPGP are relatively obscure. For instance,
OpenPGP has provisions for notarizations (
which, like meat-world notarizations, are basically signatures over
other signatures (as opposed to signatures over some data). The
following shows how they can be made and verified using
$ sq key generate -u 'Alice <firstname.lastname@example.org>' -e alice.asc --rev-cert alice.rev $ sq key generate -u 'Bob (Notar) <email@example.com>' -e bob.asc --rev-cert bob.rev $ echo 'I hereby authorize...' | sq sign --secret-key-file alice.asc > alice.msg $ sq sign --notarize --secret-key-file bob.asc <alice.msg >bob.msg $ sq verify --public-key-file bob.asc --public-key-file alice.asc bob.msg Good level 1 notarization from 0DD7 D4A7 AFD4 ECD8 Good signature from DB60 514B AC9E 68E0 I hereby authorize... 2 good signatures.
Currently, we’re not aware of any other OpenPGP implementation that supports notarizations. As such, no one is likely to fault us for ignoring it. But, given that GnuPG has recently been approved for making notarizations in Washington, US, this feature could eventually prove useful.
Nevertheless, for us, the more important reason to support these
arcane features is not somehow being able to brag about feature
completeness, but that we’ve observed that implementing such
functionality improves the overall code quality; it forces us to think
about corner cases both at the implementation level, and in terms of
the abstractions that we expose to our users. In
instance, handling notarizations is straightforward: Sequoia models
OpenPGP messages as a series of layers, and notarizations (and
meta-notarizations) are just another layer in addition to the typical
signature, encryption, and compression layers.
Other functionality that we haven’t implemented yet, but plan to,
includes support for User IDs that are not self-signed. Although such
User IDs are currently unused in the OpenPGP ecosystem, they are
allowed by the standard, and we plan to use them to help capture
statements like “Alice said to use this key for
a machine readable way. We also haven’t yet implemented certification
revocations, which are useful when you
mistakenly sign someone’s key, nor have we implemented signature
time stamps, which are needed to support third-party
time stamping services.
One part of the C bindings that we’ve invested a lot of resources into is ensuring type safety. To do this, whenever we hand out a Rust object to C—whether we transfer ownership of the object to C or just return a reference—we wrap the object. Then, when C uses the object, we check not only that the object has the expected type, but that the reference is still valid (e.g., hasn’t been freed or transferred). This functionality has already been helpful to us, and we are arguably in a better position to understand Sequoia’s API than most consumers of the library will be. This work was partially inspired by Yiming Jing’s presentation One Thousand Ways to Die in Rust FFI at the last Rust Fest in Rome.
The wrapper objects are currently realized using a bunch of macros, which, unfortunately, are rather Sequoia specific. It’s unclear if this is inevitable or whether they can be usefully abstracted. But, as we hope to improve the type safety by storing type tags in the pointer, and we want to automatically generate more C code, in particular, we want to automatically convert between idiomatic Rust and idiomatic C types, and automatically produce C headers, we’ll return to this issue later.
In writing Sequoia, our focus has been firstly on the needs of the OpenPGP ecosystem. For us, that means making sure that Sequoia’s API is easy to use.
To verify our libraries’ ease of use, we’ve been actively consuming
our own API. This firstly occurs in the context of
command-line front end to our libraries and services. But we’ve also
been working on new tools, and adapting existing tools to use Sequoia.
We already mentioned that we’ve developed a port of p≡p to Sequoia
using our C bindings. At FOSDEM, we helped Guilhem Moulin start
porting caff, a key signing tool, to Sequoia. Additionally, we’ve
worked on a Key Server written in Rust called Hagrid. And, we’ve
begun work on a Kuvert replacement called [Umschlagend], which is a
tool to automatically encrypt outgoing mail. It’s particularly useful
for putting in front of something like a Wiki instance, which sends
notifications that should be encrypted.
We are particularly pleased to see growing interest in Sequoia from the community. We regularly have people visit our irc channel (#sequoia on Freenode) to ask questions. Some people have even reported issues. And, others have asked about using Sequoia in their project.
Pijul is Sequoia’s first user. Pijul is a new simple, distributed, and fast version control system. They integrated Sequoia support so that they can start signing commits, and were quite happy with Sequoia so far:
I know from first-hand experience that implementing PGP is hard, and they are doing great work.
Hagrid, our verifying key server, originally started as a toy project to evaluate Sequoia’s API, but it has now been embraced by the broader community. In particular, Vincent Breitmoser from Open Keychain and K9 has taken an active role first in encouraging us to turn Hagrid into a working, polished product, and now more or less leading the development. Also, Nico Weichbrodt has been working on deployment. More information about this project will be forthcoming in the near future™.
dkg from Debian and the ACLU has resumed work on
packaging Sequoia for Debian. The major blocker was the lack of a
Debian package for
bindgen. That’s since been rectified, and dkg
has started to package Sequoia’s dependencies.
In addition to testing our API, one of our goals in replementing this functionality was to more faithfully reproduce the OpenPGP message’s structure. GnuPG, for instance, doesn’t show how packets are nested, and, for technical reasons, doesn’t show MDC packets. We also wanted to make the output a bit less arcane by using fewer magic numbers.
Here you can see the different between
gpg --list-packets and
sq packet dump
on the same signed and encrypted message:
$ gpg --list-packets msg.asc gpg: encrypted with 2048-bit RSA key, ID 0x08CC70F8D8CC765A, created 2017-07-19 "Justus Winter <firstname.lastname@example.org>" gpg: encrypted with 2048-bit RSA key, ID 0xC2B819056C652598, created 2015-04-07 "Neal H. Walfield <email@example.com>" # off=0 ctb=85 tag=1 hlen=3 plen=268 :pubkey enc packet: version 3, algo 1, keyid 08CC70F8D8CC765A data: [2047 bits] # off=271 ctb=85 tag=1 hlen=3 plen=268 :pubkey enc packet: version 3, algo 1, keyid C2B819056C652598 data: [2047 bits] # off=542 ctb=d2 tag=18 hlen=2 plen=0 partial new-ctb :encrypted data packet: length: unknown mdc_method: 2 # off=563 ctb=a3 tag=8 hlen=1 plen=0 indeterminate :compressed packet: algo=1 # off=565 ctb=90 tag=4 hlen=2 plen=13 :onepass_sig packet: keyid 872AA58E2C282883 version 3, sigclass 0x00, digest 10, pubkey 1, last=1 # off=580 ctb=cb tag=11 hlen=2 plen=10 new-ctb :literal data packet: mode b (62), created 1559724604, name="", raw data: 4 bytes # off=592 ctb=89 tag=2 hlen=3 plen=563 :signature packet: algo 1, keyid 872AA58E2C282883 version 4, created 1559724604, md5len 0, sigclass 0x00 digest algo 10, begin of digest 02 03 hashed subpkt 33 len 21 (issuer fpr v4 E4C0E91A5C420A2DACCFCD69872AA58E2C282883) hashed subpkt 2 len 4 (sig created 2019-06-05) subpkt 16 len 8 (issuer key ID 872AA58E2C282883) data: [4096 bits]
$ sq packet dump --session-key 2EC1E09751E34E94C83EA02CA9181EC2857B39750D793BF7F4DF914F5BDE3A9F msg.asc Old CTB, 268 bytes: Public-key Encrypted Session Key Packet Version: 3 Recipient: 08CC 70F8 D8CC 765A Pk algo: RSA (Encrypt or Sign) Old CTB, 268 bytes: Public-key Encrypted Session Key Packet Version: 3 Recipient: C2B8 1905 6C65 2598 Pk algo: RSA (Encrypt or Sign) New CTB, partial length, 512 bytes in first chunk: Encrypted and Integrity Protected Data Packet │ Version: 1 │ Session key: 2EC1E09751E34E94C83EA02CA9181EC2857B39750D793BF7F4DF914F5BDE3A9F │ Symmetric algo: AES with 256-bit key │ Decryption successful │ ├── Old CTB, indeterminate length: Compressed Data Packet │ │ Algorithm: ZIP │ │ │ ├── Old CTB, 13 bytes: One-Pass Signature Packet │ │ Version: 3 │ │ Type: Binary │ │ Pk algo: RSA (Encrypt or Sign) │ │ Hash algo: SHA512 │ │ Issuer: 872A A58E 2C28 2883 │ │ Last: true │ │ │ ├── New CTB, 10 bytes: Literal Data Packet │ │ Format: Binary data │ │ Timestamp: 2019-06-05T08:50 │ │ Content: "foo\n" │ │ │ └── Old CTB, 563 bytes: Signature Packet │ Version: 4 │ Type: Binary │ Pk algo: RSA (Encrypt or Sign) │ Hash algo: SHA512 │ Hashed area: │ Issuer Fingerprint: E4C0 E91A 5C42 0A2D ACCF CD69 872A A58E 2C28 2883 │ Signature creation time: 2019-06-05T08:50 │ Unhashed area: │ Issuer: 872A A58E 2C28 2883 │ Hash prefix: 0203 │ Level: 0 (signature over data) │ └── New CTB, 20 bytes: Modification Detection Code Packet Hash: FAB619D8E1BB8AE77CFA78432B80DC3DFFBC749A Computed hash: FAB619D8E1BB8AE77CFA78432B80DC3DFFBC749A
(For even more magic, try passing the
--hex option to
Although you still need to be familiar with OpenPGP to understand the
sq packet dump, we realized that it is probably also a
good learning tool. As such, to make it more accessible, we built a
website, dump.sequoia-pgp.org, which wraps the functionality of
dump packet. And, we also added some examples and explanations.
Check it out and let us know what you think! Also, be sure not to miss Under the Hood, the start of an introduction to OpenPGP for people who want to understand how OpenPGP works, but don’t want to read the RFC.
Unless you’ve been following our git repository, you probably haven’t noticed that we’ve been making a release about once a month. We plan to continue in this manner until the low-level library is feature complete and the API has stabilized. With respect to the API, there are still a few rough edges that we want to improve, and we also want to fix some some functionality that dictates too much policy.
Once we are done¸ we’ll make a 1.0 release of the low-level OpenPGP library and the C bindings. We hope that will happen by the end of the summer.
At the beginning of the year, Kai decided to focus his energy on a new project at 9elements.
We spent several months looking for a replacement. The most important
qualification for the three of us wasn’t someone who had 40 years of
experience working with OpenPGP and 20 years of experience programming
Rust, but a creative problem solver who can work independently. We
found several excellent people, and in the end we decided together to
juga has worked on some Tor projects, in particular, the
bandwidth scanner. Also, as part of a Prototype Fund project,
they developed a privacy-preserving DHCP client, dhcpcanon.
Although they are new to Rust, they have some prior experience working
with OpenPGP in the context of Autocrypt.
juga started working with us in mid-April. Although Sequoia has
already become a rather intimidating code base, and
juga has had to
learn Rust from scratch more or less by themself, we’ve already
integrated several of their patches.