sq user testing results

By Lars | August 28, 2022

I did some user testing of sq with five volunteers. This blog post is a report of what I learned. Good news: everyone did get all the tasks done successfully and within the one hour I had allocated, with plenty of time left over. Of course, there were a few things that could be improved.

The test setup is that I watched them perform the tasks in the plan:

  • Log into the virtual machine.
  • Generate a new key.
  • Share the certificate for the new key with me.
  • Get my certificate.
  • Certify my key and share the result with me.
  • Encrypt a message using my certificate, and send the encrypted message to me
  • Receive a response from me. Verify and decrypt the response.

Specific good points about sq I noticed:

  • Most people looked at manual pages and help output, without prompting. It’s good that sq follows modern Unix conventions here.
  • Shell TAB completion worked out of the box, and was appreciated.

Specific comments from the volunteers, quoted as exactly as my handwriting skills allow, so that others may bask in the warmth of accomplishment:

  • “Sequoia is clearly on the right path”
  • “Using this was confusing for like half a second after being used to gpg”
  • “Manual page is quite clear”
  • “Miles ahead of gpg in terms of usability”

However, some aspects of sq were stumbling blocks to several people:

  • The biggest issue was the terminology: “certify” and “certificate” are technically correct terms, but seem to be confusing to many. My assumption is that this is partly due to that terminology being new to those who’ve already used GnuPG, of even SSH, and partly because the words are so similar to each other. They’re also sufficiently unusual English words that a non-native speaker of English may struggle.

    • “extracting” a certificate was also a lesser stumbling block

  • Some error messages of sq are not as helpful as they could be. For example, when using sq certify and getting who certifies and who gets certified in the wrong order, the error message is “Error: Found no suitable key on [FINGERPRINT]”, which doesn’t guide the user to realizing their mistake. I saw several people make this specific mistake. It’d help if the error message said which file the problem originates from, or what User ID the problematic key has.

    • Similarly, the error message for wrong or missing User ID being certified could point the user to sq inspect, or make the fact that the error message actually does list the User IDs be more easily noticed: several people had trouble noticing them.

    • When (mistakenly) using sq verify to verify the signature of an encrypted message, the error message is “Error: Malformed Message: Malformed OpenPGP message”, which is tautological and misleading, not to say worrying. It could say that it’s an encrypted message and point the user towards sq decrypt --signer-cert.

    • We in Sequoia might want to review all the error messages sq may output for clarity and helpfulness.

  • sq certify gets three non-option arguments on the command line. This seems to be hard for people to get right. It might be worth making those be mandatory options instead so that the option names guide the user.

  • sq inspect was a little hard to spot for people, even when reading help output or manual pages. Two people tried sq info, which doesn’t do anything. Not sure what to do about that.

  • Three people tried to use sq sign to certify a key; I assume this is due to them being more familiar with the “sign a key” terminology. Not sure if it’s worth it, but sq could notice that a file that contains a certificate is being signed, and give a warning.

  • At least two people with experience of GnuPG were confused by sq not having an implicit, hidden keyring where it stores keys and certificates. I’m not sure this is a problem, as such. Once sq gains support for public and private key stores, this maybe-not-a-problem will presumably go away.

  • It’s hard to see what the actual decrypted message is in the output of sq decrypt. It might be better to mark it visually, or to not mix metadata and content in output to the terminal.

Overall, I would say sq proved empirically to be easy to use.

Note

This work is supported by a grant from the NLnet foundation from the NGI Assure fund, which is financially supported by the European Council.