I'll update this thread with my progress of the #golang code for

ssb message verification

According to the chapter on meesage formating from the protocol guide

Before signing a message it must be formatted as JSON as shown above. The formatting rules are:

• Two spaces for indentation.
• Dictionary entries and list elements each on their own line.
• Empty dictionaries appear as {} and empty lists appear as [].
• One space after the colon : for dictionary keys.
• No trailing newline.

To verify the signature, first remove the signature field from the message. Make sure the remaining message is formatted as JSON using the same formatting rules. Here the order of dictionary keys matters; the keys must be in the same order that you received them.

This rules out marshaling the message into map[string]interface{} in go, since iterating over any map will have random order. We could however, if we knew or decided on some canonical ordering, see the issue linked here for more background on msg canonicalization.

I tried the approach that @Vendan chose for his client. Defining the fields as a struct{} will maintain their order as defined in the code.

I still can't fully sync my feed with this approach but I'm not fully sure yet why. The messag keys start diverging so that latestMsg.key != newMsg.previous.

The approach I am currently trying is to use the json tokenizer to achive the formatting rules while maintaining the odering in which messages came in.

Some other here be dragons terretory yet to be discovered:

• js also uses utf16 strings, i heard..?
• the monotonic timestamps + the fact that js uses ieee754 floats for everything

Fell into the rabbit-hole of finding a good way to make (binary) test fixtures...

I now wrote a generator that makes a zip file with $seq.data and $seq.meta files inside. I wanted to use the comment field of the zip standard to store the metadata at first but couldn't find a npm zip package that supports them. The latter research took more time than I would have liked....

I chose zip because I don't want to check in thousands of files or create a giant single file which also isn't readable.

I verified that this is a problem:

Defining the fields as a struct{} will maintain their order as defined in the code.

I made a tool to generate test data and the above approach breaks agains the output from ssb-feed. On this feed it happens on message seq:5049. Before the static [Previous, Author, Sequence, Timestamp, Hash, Content, Signature] works but then, seq and author somehow switched, resulting in different hashes and signatures..

This ordering issue also confirms a problem @keks suspected. We have to reproduce this ordering when gossiping feeds to other peers. For practical reasons we will just save the message as raw bytes for later, I guess. I find all this quite depressing from a design perspective but it should work out.

Coming up next: making the tests pass for the new pretty printer. Ordering won't be a problem for it but sadly the go tokenizer hides commas and colons from the user, so it needs to keep track of object/array nesting..

I made a new repo for all this: %95qJrYg...

Just posted %xw/lcPY... which gets the object and array nesting right. Timestamps also format correctly as far as this feed is enough of a test for that. I am now starting to see diffs in in utf8 characters now, not sure yet what to do about it.

@dominic: I saw a validation issues on github. I am using ssb-feed to construct my test data (because thats what the protocol guide links to) should I be using ssb-validate? How are they different.

cc @arj

@cryptix they should be the same - turns out the 11.3 bug was a different thing. I think ssb-validate is better now, and has better tests.

I pushed fixes for the last two TODOs in %DoRbVdh...

It fixes the problems of empty obj/array printing and solves the escaped unicode sequences with a regexp replace.

I would have liked not to involve a regexp but couldn't find a way that also preserves the newlines as \n.

The code now passes for all the 6400 messages on this feed and I will now focus on removing the signature field and doing the verification.
Now that I have a matching encoding, I don't have large doubts about this.

verification

works as of %ycn2bkf...

PHEEEEEEW!!!!! I really feard doing this and am really glad it is done (in approx two days of work).

I small bug creepd in the unicode replace. The regexp has {4,8} as a quiantifier which means 4,5,6,7,8 not 4 or 8.
I need to expand that behavior. I have a feed where the regexp captures too much and thus invalidates the message.

I pushed a better approach to the formatting problems. I doesn't rely on golangs %q to quote and escape strings since this introduced the problem with re-encoding unicode characters.

Instead of %q+regexp for unicode replacement it now uses plain string.Replace to encode \n, \t, \r, \\ and \" correctly. We still need to encode some non-printable control characters as \uXXXX but I've only seen one instance of this yet. All the other feeds I had problems with now validate fully.

on newly found s side.. there is this:

handleConnect="fetchFeed stored failed" err="fetchFeed(@TXKFQehlyoSn8UJAIVP/k2BjFINC591MlBC2e2d24mA=.ed25519:1556): message verify failed: ssb Verify(@TXKFQehlyoSn8UJAIVP/k2BjFINC591MlBC2e2d24mA=.ed25519:1557): could not verify message: sbot: invalid signature"

err="fetchFeed(@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519:4505): message verify failed: ssb Verify(@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519:4506): could not verify message: sbot: invalid signature"

i'm not sure what is going on there yet..

cc @keks

@william kind agree..

there were some weirdnesses in our code and I didn't see it again yet but also really not looking forward to it...