At the Tree Nursery, First Impressions

By Neal | June 6, 2019

Our last status update was six months ago, shortly after our first preliminary release. Since then, quite a bit has happened in and around Sequoia.

Sequoia Development

For us, Sequoia is more than just a new OpenPGP implementation. Sequoia is firstly a project to rethink the OpenPGP ecosystem. We are convinced that OpenPGP has a lot of untapped potential. In particular, we think its flexible support for modeling trust (the web of trust is just one trust model built on top of OpenPGP’s certification mechanisms, which never had good tooling for advanced users never mind normal people), and the ability to update encryption & signature keys, and identity information without changing the key’s primary identifier are underappreciated and under used features.

To this end, we decided our first goal would be a low-level, policy-free library. And, after we nailed that, we’d work out a high-level, opinionated API, which covers common use cases. Our overarching goal is to make it easier to add support for OpenPGP to applications in a safe manner while allowing developers to override those high-level decisions that are inappropriate for their application in a straightforward fashion with minimal effort.

Currently, we are completing and polishing Sequoia’s low-level API and the corresponding C bindings. By the end of the summer, we hope that we’ll have gotten enough feedback that we can declare it stable, make a 1.0 release, and concentrate on the higher-level API and accompanying services.

The Low-Level Library

An important technical goal for Sequoia’s low-level library is to not simply implement the OpenPGP standard, but to expose a safe, easy-to-use, and policy-free interface. This means, for instance, that when an application signs a message with an OpenPGP key that has multiple signing-capable subkeys, our low-level library doesn’t just choose one of them, but makes it easy for the caller to select the right one for the task at hand.

A low-level library sounds like it would be hard to use. But our experience porting the p≡p cryptographic engine to Sequoia is that the PGP support requires less code using Sequoia than using GPGME 1, the official high-level interface to GnuPG, which the p≡p engine currently uses.

At this point, we feel like the low-level library implements about 99% of the OpenPGP standard. You can find details about what we’ve implemented, and what we haven’t as well as some caveats on our status page.

The last few percent of OpenPGP are relatively obscure. For instance, OpenPGP has provisions for notarizations (signature nesting), which, like meat-world notarizations, are basically signatures over other signatures (as opposed to signatures over some data). The following shows how they can be made and verified using sq:

$ sq key generate -u 'Alice <alice@example.org>' -e alice.asc --rev-cert alice.rev
$ sq key generate -u 'Bob (Notar) <bob@example.com>' -e bob.asc --rev-cert bob.rev
$ echo 'I hereby authorize...' | sq sign --secret-key-file alice.asc > alice.msg
$ sq sign --notarize --secret-key-file bob.asc <alice.msg >bob.msg
$ sq verify --public-key-file bob.asc --public-key-file alice.asc bob.msg
Good level 1 notarization from 0DD7 D4A7 AFD4 ECD8
Good signature from DB60 514B AC9E 68E0
I hereby authorize...
2 good signatures.

Currently, we’re not aware of any other OpenPGP implementation that supports notarizations. As such, no one is likely to fault us for ignoring it. But, given that GnuPG has recently been approved for making notarizations in Washington, US, this feature could eventually prove useful.

Nevertheless, for us, the more important reason to support these arcane features is not somehow being able to brag about feature completeness, but that we’ve observed that implementing such functionality improves the overall code quality; it forces us to think about corner cases both at the implementation level, and in terms of the abstractions that we expose to our users. In sq, for instance, handling notarizations is straightforward: Sequoia models OpenPGP messages as a series of layers, and notarizations (and meta-notarizations) are just another layer in addition to the typical signature, encryption, and compression layers.

Other functionality that we haven’t implemented yet, but plan to, includes support for User IDs that are not self-signed. Although such User IDs are currently unused in the OpenPGP ecosystem, they are allowed by the standard, and we plan to use them to help capture statements like “Alice said to use this key for bob@example.org” in a machine readable way. We also haven’t yet implemented certification revocations, which are useful when you mistakenly sign someone’s key, nor have we implemented signature time stamps, which are needed to support third-party time stamping services.

C Bindings

The C bindings don’t yet cover all of the low-level API, but they do cover the most important functionality. The main driver for this has been our port of the p≡p engine to Sequoia.

One part of the C bindings that we’ve invested a lot of resources into is ensuring type safety. To do this, whenever we hand out a Rust object to C—whether we transfer ownership of the object to C or just return a reference—we wrap the object. Then, when C uses the object, we check not only that the object has the expected type, but that the reference is still valid (e.g., hasn’t been freed or transferred). This functionality has already been helpful to us, and we are arguably in a better position to understand Sequoia’s API than most consumers of the library will be. This work was partially inspired by Yiming Jing’s presentation One Thousand Ways to Die in Rust FFI at the last Rust Fest in Rome.

The wrapper objects are currently realized using a bunch of macros, which, unfortunately, are rather Sequoia specific. It’s unclear if this is inevitable or whether they can be usefully abstracted. But, as we hope to improve the type safety by storing type tags in the pointer, and we want to automatically generate more C code, in particular, we want to automatically convert between idiomatic Rust and idiomatic C types, and automatically produce C headers, we’ll return to this issue later.

API Testing

In writing Sequoia, our focus has been firstly on the needs of the OpenPGP ecosystem. For us, that means making sure that Sequoia’s API is easy to use.

To verify our libraries’ ease of use, we’ve been actively consuming our own API. This firstly occurs in the context of sq, our command-line front end to our libraries and services. But we’ve also been working on new tools, and adapting existing tools to use Sequoia. We already mentioned that we’ve developed a port of p≡p to Sequoia using our C bindings. At FOSDEM, we helped Guilhem Moulin start porting caff, a key signing tool, to Sequoia. Additionally, we’ve worked on a Key Server written in Rust called Hagrid. And, we’ve begun work on a Kuvert replacement called Umschlagend 2, which is a tool to automatically encrypt outgoing mail. It’s particularly useful for putting in front of something like a Wiki instance, which sends notifications that should be encrypted.

Community

We are particularly pleased to see growing interest in Sequoia from the community. We regularly have people visit our irc channel (#sequoia on Freenode) to ask questions. Some people have even reported issues. And, others have asked about using Sequoia in their project.

Pijul is Sequoia’s first user. Pijul is a new simple, distributed, and fast version control system. They integrated Sequoia support so that they can start signing commits, and were quite happy with Sequoia so far:

I know from first-hand experience that implementing PGP is hard, and they are doing great work.

Hagrid, our verifying key server, originally started as a toy project to evaluate Sequoia’s API, but it has now been embraced by the broader community. In particular, Vincent Breitmoser from Open Keychain and K9 has taken an active role first in encouraging us to turn Hagrid into a working, polished product, and now more or less leading the development. Also, Nico Weichbrodt has been working on deployment. More information about this project will be forthcoming in the near future(tm).

dkg from Debian and the ACLU has resumed work on packaging Sequoia for Debian. The major blocker was the lack of a Debian package for bindgen. That’s since been rectified, and dkg has started to package Sequoia’s dependencies.

anarcat, a co-author of Monkeysign, recently discovered Sequoia. His first order of business was to create a Docker file for trying Sequoia.

dump.sequoia-pgp.org

It’s sometimes helpful to inspect an OpenPGP message. The main tools for this are gpg --list-packets and pgpdump. We reimplemented this functionality in sq, Sequoia’s command-line front end.

In addition to testing our API, one of our goals in replementing this functionality was to more faithfully reproduce the OpenPGP message’s structure. GnuPG, for instance, doesn’t show how packets are nested, and, for technical reasons, doesn’t show MDC packets. We also wanted to make the output a bit less arcane by using fewer magic numbers.

Here you can see the different between gpg --list-packets and sq packet dump on the same signed and encrypted message:

$ gpg --list-packets msg.asc
gpg: encrypted with 2048-bit RSA key, ID 0x08CC70F8D8CC765A, created 2017-07-19
      "Justus Winter <justus@sequoia-pgp.org>"
gpg: encrypted with 2048-bit RSA key, ID 0xC2B819056C652598, created 2015-04-07
      "Neal H. Walfield <neal@walfield.org>"
# off=0 ctb=85 tag=1 hlen=3 plen=268
:pubkey enc packet: version 3, algo 1, keyid 08CC70F8D8CC765A
	data: [2047 bits]
# off=271 ctb=85 tag=1 hlen=3 plen=268
:pubkey enc packet: version 3, algo 1, keyid C2B819056C652598
	data: [2047 bits]
# off=542 ctb=d2 tag=18 hlen=2 plen=0 partial new-ctb
:encrypted data packet:
	length: unknown
	mdc_method: 2
# off=563 ctb=a3 tag=8 hlen=1 plen=0 indeterminate
:compressed packet: algo=1
# off=565 ctb=90 tag=4 hlen=2 plen=13
:onepass_sig packet: keyid 872AA58E2C282883
	version 3, sigclass 0x00, digest 10, pubkey 1, last=1
# off=580 ctb=cb tag=11 hlen=2 plen=10 new-ctb
:literal data packet:
	mode b (62), created 1559724604, name="",
	raw data: 4 bytes
# off=592 ctb=89 tag=2 hlen=3 plen=563
:signature packet: algo 1, keyid 872AA58E2C282883
	version 4, created 1559724604, md5len 0, sigclass 0x00
	digest algo 10, begin of digest 02 03
	hashed subpkt 33 len 21 (issuer fpr v4 E4C0E91A5C420A2DACCFCD69872AA58E2C282883)
	hashed subpkt 2 len 4 (sig created 2019-06-05)
	subpkt 16 len 8 (issuer key ID 872AA58E2C282883)
	data: [4096 bits]
$ sq packet dump --session-key 2EC1E09751E34E94C83EA02CA9181EC2857B39750D793BF7F4DF914F5BDE3A9F msg.asc 
Old CTB, 268 bytes: Public-key Encrypted Session Key Packet
    Version: 3
    Recipient: 08CC 70F8 D8CC 765A
    Pk algo: RSA (Encrypt or Sign)
  
Old CTB, 268 bytes: Public-key Encrypted Session Key Packet
    Version: 3
    Recipient: C2B8 1905 6C65 2598
    Pk algo: RSA (Encrypt or Sign)
  
New CTB, partial length, 512 bytes in first chunk: Encrypted and Integrity Protected Data Packet
│   Version: 1
│   Session key: 2EC1E09751E34E94C83EA02CA9181EC2857B39750D793BF7F4DF914F5BDE3A9F
│   Symmetric algo: AES with 256-bit key
│   Decryption successful
│ 
├── Old CTB, indeterminate length: Compressed Data Packet
│   │   Algorithm: ZIP
│   │ 
│   ├── Old CTB, 13 bytes: One-Pass Signature Packet
│   │       Version: 3
│   │       Type: Binary
│   │       Pk algo: RSA (Encrypt or Sign)
│   │       Hash algo: SHA512
│   │       Issuer: 872A A58E 2C28 2883
│   │       Last: true
│   │     
│   ├── New CTB, 10 bytes: Literal Data Packet
│   │       Format: Binary data
│   │       Timestamp: 2019-06-05T08:50
│   │       Content: "foo\n"
│   │     
│   └── Old CTB, 563 bytes: Signature Packet
│           Version: 4
│           Type: Binary
│           Pk algo: RSA (Encrypt or Sign)
│           Hash algo: SHA512
│           Hashed area:
│             Issuer Fingerprint: E4C0 E91A 5C42 0A2D ACCF  CD69 872A A58E 2C28 2883
│             Signature creation time: 2019-06-05T08:50
│           Unhashed area:
│             Issuer: 872A A58E 2C28 2883
│           Hash prefix: 0203
│           Level: 0 (signature over data)
│         
└── New CTB, 20 bytes: Modification Detection Code Packet
        Hash: FAB619D8E1BB8AE77CFA78432B80DC3DFFBC749A
        Computed hash: FAB619D8E1BB8AE77CFA78432B80DC3DFFBC749A
      

(For even more magic, try passing the --hex option to sq packet dump!)

Although you still need to be familiar with OpenPGP to understand the output of sq packet dump, we realized that it is probably also a good learning tool. As such, to make it more accessible, we built a website, dump.sequoia-pgp.org, which wraps the functionality of sq dump packet. And, we also added some examples and explanations.

Check it out and let us know what you think! Also, be sure not to miss Under the Hood, the start of an introduction to OpenPGP for people who want to understand how OpenPGP works, but don’t want to read the RFC.

Releases?

Unless you’ve been following our git repository, you probably haven’t noticed that we’ve been making a release about once a month. We plan to continue in this manner until the low-level library is feature complete and the API has stabilized. With respect to the API, there are still a few rough edges that we want to improve, and we also want to fix some some functionality that dictates too much policy.

Once we are done¸ we’ll make a 1.0 release of the low-level OpenPGP library and the C bindings. We hope that will happen by the end of the summer.

Kai leaves, juga comes

At the beginning of the year, Kai decided to focus his energy on a new project at 9elements.

We spent several months looking for a replacement. The most important qualification for the three of us wasn’t someone who had 40 years of experience working with OpenPGP and 20 years of experience programming Rust, but a creative problem solver who can work independently. We found several excellent people, and in the end we decided together to hire juga.

juga has worked on some Tor projects, in particular, the bandwidth scanner. Also, as part of a Prototype Fund project, they developed a privacy-preserving DHCP client, dhcpcanon. Although they are new to Rust, they have some prior experience working with OpenPGP in the context of Autocrypt.

juga started working with us in mid-April. Although Sequoia has already become a rather intimidating code base, and juga has had to learn Rust from scratch more or less by themself, we’ve already integrated several of their patches.


  1. Update 2023: pEpEngine sources moved to a self-hosted gitea. ↩︎

  2. Update 2023: Project Umschlagend has been renamed to koverto ↩︎