Thoughts on To Sign or Not to Sign

39C3, the annual meeting of the Chaos Computer Club (CCC), included a presentation called To sign or not to sign: Practical vulnerabilities in GPG & friends. In their presentation, the security researchers discuss the vulnerabilities that they found in GnuPG, Sequoia, age and minisign. The talk is impressive not the least for the shear number of vulnerabilities (14!) that they found, but also their breadth. They range from buffer overflows, to the use of uninitialized memory, to improper input validation.

In this blog post, I will take a look at the attack that the researchers claim demonstrates a security weakness in Sequoia, and consider its possible impact. In my estimation, this characterization is primarily due to a literal translation of gpg invocations to sq invocations, and the user ignoring sq’s output. As the user is following a recipe, a more realistic analysis should have considered a less naive translation that uses sq’s standard workflows, which would have prevented the attack. That said, the security researchers identify an issue that raises legitimate concerns, and the ecosystem as a whole needs to improve to better protect users.

Security Researchers

Before talking about the presentation, I want to take a moment to commend researchers who look for vulnerabilities and responsibly report them.

This type of work is often thankless. Researchers invest a lot of time auditing programs without finding any issues. These negative results are frequently viewed as little better than no results. Indeed, bug bounty programs are based on this notion: only new vulnerabilities are worthy of a reward; confirmation that a class of vulnerabilities was searched for, but not found is not. This perspective is reinforced by news organizations who give more attention to vulnerabilities than clear bills of health. Further, when a vulnerability is found, it is easy to view the researcher as an adversary instead of a collaborator.

Third-parties who audit code with fresh eyes, and different perspectives are essential for identifying latent issues, and discovering novel attacks. This is critical to ensuring software is hardened before the bad guys discover the vulnerabilities and leverage them for malicious purposes.

Signature Verification Attack

The researchers open their presentation by demonstrating an impressive attack on signature verification. They go through the documented steps of downloading Fedora, verifying the digital signature, and finally booting what is clearly not the official Fedora release.

They primarily demonstrate the attack using GnuPG, but for one step, they indicate that Sequoia exhibits the same behavior. This is where I think we need to carefully examine what mechanisms sq already provides to mitigate the problem, and why it’s important to be clear about how verification is supposed to take place with each tool; sq is not gpg, and it provides means to detect the attack when used properly, as I’ll explain below.

The Setting

Before we examine the attack, let’s first consider the setting.

When a release manager releases a piece of software, they often sign the software. The signature allows someone to verify that the software they downloaded is what the release manager intended even if they got the software from an untrusted third party. To verify the signature, the user needs to have the release manager’s certificate, and software that can verify the signature. In the FOSS world, OpenPGP is often used.

An alternative approach that provides somewhat weaker security guarantees is the release manager publishes a file containing the checksum of the released files on a trusted server. The user can still obtain the software from a potentially untrusted third party, but they can verify it using the small checksum file that they obtain from the trusted server. This approach avoids the problem of having to find and verify the release manager’s certificate, and uses much simpler software.

To allow the user to choose which method they prefer, some projects support both. One way to achieve this is to sign the checksum file. This adds an extra step when verifying the signature, but has the nice property that only one artifact is signed per release. The verification process is then:

1. Download the software
2. Download the signed checksum file
3. Download and authenticate the release manager’s certificate
4. Verify the signature
5. Verify the checksum
6. Use the software

If the user doesn’t want to verify the signature, they can skip the middle two steps. The release manager’s certificate also only needs to be authenticated the first time it is used.

The above steps have severe security implications and can easily be done wrong. This is true whether using OpenPGP or some other system like minisign. Providing clear and accessible documentation is a essential.

An important bit to note is that the checksum file contains both the checksums and the digital signature. Here’s a slightly abridged example of a signed checksum file:

 1-----BEGIN PGP SIGNED MESSAGE-----
 2Hash: SHA256
 3
 4# Fedora-Workstation-Live-43-1.6.x86_64.iso: 2742190080 bytes
 5SHA256 (Fedora-Workstation-Live-43-1.6.x86_64.iso) = 2a4a16c009244eb5ab2198700eb04103793b62407e8596f30a3e0cc8ac294d77
 6-----BEGIN PGP SIGNATURE-----
 7
 8iQIzBAEBCAAdFiEExufwgc+A4TFGZ26IgptgZjFkVTEFAmj9Du0ACgkQgptgZjFk
 9...
10XsRsb5zuX9c3iN+zmLLn4kQ0bmO228bhjqMctIe45dK1DCa0KlA=
11=rBWY
12-----END PGP SIGNATURE-----

The fifth line contains the name of the file and the checksum. The lines between BEGIN PGP SIGNATURE and END PGP SIGNATURE contain the digital signature over lines 4 and 5. In OpenPGP, this is called a cleartext signature, because the signed data is directly readable by a human. OpenPGP supports two other signature formats: a detached signature, which is a signature over a file, and an inline signature, which is a signature over data in the same file. In the latter case, the data is also encoded as OpenPGP data, and not human readable. In general, it is possible to convert a signed artifact from one format to another without breaking the signature. That is, we can take a cleartext signature and convert it into an inline signature. This also works in the opposite direction if the signed data does not include control characters.

The Attack

Let’s now look at the attack.

Fedora publishes a signed checksum file and the following instructions on how to verify it:

Verify your download

Verify your download for security and integrity using the proper checksum file. If there is a good signature from one of the Fedora keys, and the SHA256 checksum matches, then the download is valid.

1. Download the checksum file into the same directory as the image you downloaded.

2. Import Fedora’s GPG key(s)

curl -O https://fedoraproject.org/fedora.gpg

You can verify the details of the GPG key(s) here.

3. Verify the checksum file is valid

gpgv --keyring ./fedora.gpg Fedora-Workstation-43-1.6-x86_64-CHECKSUM

4. Verify the checksum matches

sha256sum --ignore-missing -c Fedora-Workstation-43-1.6-x86_64-CHECKSUM

If the output states that the file is valid, then it’s ready to use!

In the attack shown in the presentation, the attackers replace the software (Fedora-Workstation-Live-43-1.6.x86_64.iso) and the signed checksum file (Fedora-Workstation-43-1.6-x86_64-CHECKSUM). Normally, this would mean that even though the checksum matches, the digital signature would fail to verify, because it was not created by the Fedora signing certificate, and disaster would be avoided.

To get around the last problem, the attackers don’t generate a new signature. Instead, using a tool they developed, they convert the original signature over the correct checksum into an inline signature over the correct checksum, and they replace the cleartext part with their malicious checksum. That is, they convert:

 1-----BEGIN PGP SIGNED MESSAGE-----
 2
 3SHA256 (Fedora-Workstation-Live-43-1.6.x86_64.iso) = INTENDED CHECKSUM
 4-----BEGIN PGP SIGNATURE-----
 5
 6iQIz... \
 7...      | Signature over the above cleartext
 8b5zu...  |
 9=rBWY   /
10-----END PGP SIGNATURE-----

To:

 1-----BEGIN PGP SIGNED MESSAGE-----
 2
 3SHA256 (Fedora-Workstation-Live-43-1.6.x86_64.iso) = MALICIOUS CHECKSUM
 4-----BEGIN PGP SIGNATURE-----
 5
 6iQIz... \
 7...      | Signature over
 8XsRs...  | SHA256 (Fedora-Workstation-Live-43-1.6.x86_64.iso) = INTENDED CHECKSUM
 9=sAX2   /
10-----END PGP SIGNATURE-----

Now, when gpg or sq verifies the signature, they verify the inline signature and the data in the base64 block and ignore the cleartext data. Since the signature is correct for that data, they both indicate that and exit with success. sha256sum then verifies the checksum. It ignores the OpenPGP signature and the original checksum, which it can’t see, and checks that the file matches the malicious checksum in the cleartext data. Since the malicious checksum matches, it returns success! The issue is that sha256sum does not use the checksum that gpg and sq verified.

Note that an attacker doesn’t even need to go to so much trouble. They could just add a new file before the clear-signed message like so:

 1SHA256 (Fedora-Workstation-Live-43-1.7.x86_64.iso) = MALICOUS CHECKSUM
 2
 3-----BEGIN PGP SIGNED MESSAGE-----
 4
 5SHA256 (Fedora-Workstation-Live-43-1.6.x86_64.iso) = INTENDED CHECKSUM
 6-----BEGIN PGP SIGNATURE-----
 7
 8iQIz... \
 9...      | Signature over the above cleartext
10b5zu...  |
11=rBWY   /
12-----END PGP SIGNATURE-----

When verifying checksums, sha256sum will consider both the line before the signature and the one after. Admittedly this attack is more obvious, but relying on the user to notice this discrepancy is a poor defense.

Analysis

The primary reason for the security failure is that sha256sum works on the original file and consequently uses the unverified, malicious checksum to verify the file. The most important fix is to update the verification instructions to change sha256sum to use the verified data.

To help prevent the user from making this mistake, sq outputs the verified text to stdout. If you are very attentive, you can see this in the presentation. Unfortunately, the sq invocation is only shown for three seconds, and the presenters don’t mention it. It would be useful for future presentations if the sq invocation were shown for slightly longer and the discrepancy and sq’s preferred workflow be discussed in context.

So instead of running:

sq verify --cleartext --signer-file ./fedora.gpg Fedora-Workstation-43-1.6-x86_64-CHECKSUM
sha256sum --ignore-missing -c Fedora-Workstation-43-1.6-x86_64-CHECKSUM

The user should pass the verified data to sha256sum as follows:

sq verify --cleartext --signer-file ./fedora.gpg Fedora-Workstation-43-1.6-x86_64-CHECKSUM \
  | sha256sum --ignore-missing -c -

And in their write up, this is what the security researchers suggest:

To prevent confusion about the actual signed data, OpenPGP implementations should output the data bound by the signature during validation by default. sequoia-sq does so.

Sequoia is thus already following their recommendations.

A Real Bug

There is a real bug in Sequoia. When verifying a signature using sq, the caller specifies the type of signature that should be checked. In this case, we use --cleartext. Yet, the inline signature was verified, which should only be done if the caller passed --message. This is due to a known issue in our library, which unfortunately we haven’t yet had the chance to fix. Had we fixed this, this would have mitigated this attack. Nevertheless, the possibility for confusion remains, and the next step should always use the verified data and not the original file. We plan to address this issue this quarter. Thanks to the security researchers for showing us that the issue has a practical security impact.

Doing Even Better

There are few things that Sequoia does and can do to improve the verification workflow. First, sq download downloads a file and a signature and only outputs verified data. This simplifies the workflow as follows:

sq network search https://fedoraproject.org/fedora.gpg
sq pki link add --cert C6E7F081CF80E13146676E88829B606631645531 --all
sq download \
  --url https://download.fedoraproject.org/../Fedora-Workstation-43-1.6-x86_64-CHECKSUM \
  --cleartext \
  --output Fedora-Workstation-43-1.6-x86_64-CHECKSUM
sha256sum --ignore-missing Fedora-Workstation-43-1.6-x86_64-CHECKSUM

We also plan to add extend sq download and sq verify to support verifying signed checksums files directly this year, as documented in this issue.

Finally, we will reach out to projects like Fedora to help them improve their verification documentation.

Conclusion

The researchers have done impressive work, and we by no means intend to disparage that; on the contrary, they bring up a legitimate issue. Discussion of adequate mitigations for the attack is needed within the community, and we applaud them for their work.

We wish that there had been a better explanation that the signature verification attack against Sequoia requires the user to use sq incorrectly, and that sq already includes the proposed mitigation. This is clearer in the write up, but in reading the comments on the internet, it seems that sq’s approach is widely misunderstood.

As a final note, we welcome reports about vulnerabilities in Sequoia, and have a bug bounty program that rewards valid vulnerabilities. The triage team at YesWeHack also reviews the reports, and ensures that valid reports are brought to our attention.