By Lars | August 28, 2022
I did some user testing of
sq with five volunteers. This blog post
is a report of what I learned. Good news: everyone did get all the
tasks done successfully and within the one hour I had allocated, with
plenty of time left over. Of course, there were a few things that
could be improved.
The test setup is that I watched them perform the tasks in the plan:
- Log into the virtual machine.
- Generate a new key.
- Share the certificate for the new key with me.
- Get my certificate.
- Certify my key and share the result with me.
- Encrypt a message using my certificate, and send the encrypted message to me
- Receive a response from me. Verify and decrypt the response.
Specific good points about
sq I noticed:
- Most people looked at manual pages and help output, without
prompting. It’s good that
sqfollows modern Unix conventions here.
- Shell TAB completion worked out of the box, and was appreciated.
Specific comments from the volunteers, quoted as exactly as my handwriting skills allow, so that others may bask in the warmth of accomplishment:
- “Sequoia is clearly on the right path”
- “Using this was confusing for like half a second after being used to gpg”
- “Manual page is quite clear”
- “Miles ahead of gpg in terms of usability”
However, some aspects of
sq were stumbling blocks to several people:
The biggest issue was the terminology: “certify” and “certificate” are technically correct terms, but seem to be confusing to many. My assumption is that this is partly due to that terminology being new to those who’ve already used GnuPG, of even SSH, and partly because the words are so similar to each other. They’re also sufficiently unusual English words that a non-native speaker of English may struggle.
- “extracting” a certificate was also a lesser stumbling block
Some error messages of
sqare not as helpful as they could be. For example, when using
sq certifyand getting who certifies and who gets certified in the wrong order, the error message is “Error: Found no suitable key on [FINGERPRINT]”, which doesn’t guide the user to realizing their mistake. I saw several people make this specific mistake. It’d help if the error message said which file the problem originates from, or what User ID the problematic key has.
Similarly, the error message for wrong or missing User ID being certified could point the user to
sq inspect, or make the fact that the error message actually does list the User IDs be more easily noticed: several people had trouble noticing them.
When (mistakenly) using
sq verifyto verify the signature of an encrypted message, the error message is “Error: Malformed Message: Malformed OpenPGP message”, which is tautological and misleading, not to say worrying. It could say that it’s an encrypted message and point the user towards
sq decrypt --signer-cert.
We in Sequoia might want to review all the error messages
sqmay output for clarity and helpfulness.
sq certifygets three non-option arguments on the command line. This seems to be hard for people to get right. It might be worth making those be mandatory options instead so that the option names guide the user.
sq inspectwas a little hard to spot for people, even when reading help output or manual pages. Two people tried
sq info, which doesn’t do anything. Not sure what to do about that.
Three people tried to use
sq signto certify a key; I assume this is due to them being more familiar with the “sign a key” terminology. Not sure if it’s worth it, but
sqcould notice that a file that contains a certificate is being signed, and give a warning.
At least two people with experience of GnuPG were confused by
sqnot having an implicit, hidden keyring where it stores keys and certificates. I’m not sure this is a problem, as such. Once
sqgains support for public and private key stores, this maybe-not-a-problem will presumably go away.
It’s hard to see what the actual decrypted message is in the output of
sq decrypt. It might be better to mark it visually, or to not mix metadata and content in output to the terminal.
Overall, I would say
sq proved empirically to be easy to use.
This work is supported by a grant from the NLnet foundation from the NGI Assure fund, which is financially supported by the European Council.