Dark Crystal User Testing

cc: #darkcrystal #dark-crystal #mmt #dark-crystal-research

Test case wished to remain anonymous, so I will refer to them as Grace.

Approach:

• Given minimal prior information
• Attempt to simulate a new user arriving after installing from our website
• 3 computers setup on same LAN
• @kieran & @kieran-two were on-hand to return shards
• tried to have users sync'ed and up to date before we started the test, but that was hard!
• Video'ed (for personal reflection)

Prior information:

• A nearly new scuttlebutt user, had installed Patchwork over the weekend but hadn't really used it at all, just had let it replicate for a while.
• Grace was not a tech nerd, but tech-savvy as a regular internet-generation kiddie

Grace was given the following instructions:

1. Download and install Patchbay

• Go to https://github.com/ssbc/patchbay/releases
• Find the latest release, donwload and install

1. Open Patchbay

• Click on the circle icon in the top-right corner and click on the menu option /dark-crystal

1. Visit DarkCrystal website

• Nagivate to https://darkcrystal.pw
• Read the description of the application on the landing page

1. Your Mission!

• Share a secret with @kieran and @kieran-two

• Recover the secret you shared

• (additional step we half-played with) return a shard belonging to @kieran

Live Observations

We had many non-UI related issues which meant that we were unable to dive too deep into discussing design.

• If Patchbay hasn't indexed / loaded fully, you can't find custodians (suggestify doesn't work properly, all the names don't appear).
• The exact case was Grace tried to create a new secret and
• @kieran wouldn't appear, only two people did were full public keys @ABCD1234... (and they weren't the right people either).
• Grace did not assume they were the same person, but actually typed in @kieran @kieran-two and tried to see if it worked.
• Errors were thrown, but the error wasn't particularly helpful / descriptive enough of how to help deal with the problem.

• After minor intervention to explain what was going wrong, Grace then assumed the two identities were the correct people. Only after further intervention to correct her and ask her to restart the app (and wait 20 minutes) were we able to continue.

• When creating a secret, Grace first tried to select 3 instead of 2 quorum. This was a test to see if it would break, rather than actually making a wrong decision

• Sharing a secret was obvious

• SSB has major onboarding problems! People expect apps to just work / people are impatient

• Shard never arrived with @kieran-two :man_shrugging: so Grace was unable to retreive the secret

• Patchbay problem - indexes constantly reset? Friends never load or are missing some people.\

• Configuration / setup issues

• make sure everyone is connected and following eachother in advance

• Shard sent to Grace by @kieran in 3rd part of mission never arrived!

• Grace observed if you have more than a certain number of secrets, how are you going to remember which one is which?! And how are you going to filter / find the correct one?

Summary

• Had to wait at least 20 minutes for Patchbay to index which was disruptive
• Shards didn't gossip properly, or weren't rendering.
• Sharing a new secret was obvious
• Custodian suggester input field should really show everyone you're connected to immediately, shouldn't show only half the number of people, etc.
• Better view for list of secrets required
• Does naming a secret work? Will you remember? Does that require a 'long-term' user test? e.g. get someone to attempt to recover one of a number of secrets they shared 3 months after they shared it.

Personal reflections

• Nothing too new, I already knew about SSB onboarding issues!
• Feels difficult having to lower people's expectations when starting the app, 'and now we've gotta wait for a while...'
• Feels difficult doing it in Patchbay as its an extra step which feels unnecessary / unwanted
• Ongoing challenge of uncertainty of replication / managing people's expectations for immediate data sharing - sometimes the data doesn't arrive in time or at all. Sometimes it arrives perfectly. Never knowing whether its going to work or not is frustrating and stressful.
• /network feature wasn't available as part of the latest release, only in git master branch, so was unable to use that new feature to streamline the process of getting Grace connected to @kieran and @kieran-two.

Lessons for Next Run!

• Will try with absolute bare minimum, get someone to use a blank identity on my computer, unconnected, and see what happens. Screen out SSB on-boarding troubles to get clear interaction with just Dark Crystal
• Won't use Patchbay, will use Dark Crystal standalone installer.
• Get someone to navigate my existing interface (with loads of dummy shards) and see if they can work out how to move around that.