You are reading content from Scuttlebutt
@Dominic %Xf2JIAYPmJJ/0SRVoVQn3NwlC636pQbXmJW1HjLIKQ4=.sha256
Re: %Pnas64BJ6

I'm looking at the code - does it store the top level values as bipf arrays? https://github.com/arj03/ssb-butt2/blob/master/index.js#L8-L11

layered? because they the content is encoded as a bipf object, but then it looks like it's encoded as a bipf buffer in another array?

Reading the code I spotted several things, that feel "nit picky" to mention and 7 times faster is great anyway, but I'll ask anyway. The main question I have is about how the batch signing works.

it also seems to me that there are micro-optimizations that could add up,
for example, https://github.com/arj03/ssb-butt2/blob/master/index.js#L288
using Object.keys to get an array of the signature keys when for(var key in signatures) might be faster?

Instead of layering, what about just implementing a non-recursive decode, it would just decode the top layer and return buffers for the values? seems it might achive the same result, while still keeping the whole thing as a single valid bipf object?

The important question: how does batch signing work?

https://github.com/arj03/ssb-butt2/blob/master/index.js#L288

does it keep in memory an array of all signatures? else why is it accessing a signature from an array based on the sequence?
https://github.com/arj03/ssb-butt2/blob/master/index.js#L256

It sounds like you use specially formatted batch signature messages?

I think you could make a much simpler approach: since hashes preserve the uniqueness of the hashed data, it means signing the hash is as good as signing the data, and signing the hash of the hash (or the hash of data containing the hash) is also good enough. So therefore, every ssb message signs not just the message content but the whole chain of preceding messages. If you changed one bit in the first message the hashes of all subsequent messages would change.

So... instead of having a special batch signature - just sign each message like normal and when verifying, queue up N messages, then randomly verify one. then you can write the messages up to the verified one.

@Anders %PM68wm2oNpNIPBE3Ck9rFfXUEaWSUg6jDJ2NxV3fJuM=.sha256

hi @Dominic. Thanks for taking a look :)

I'm looking at the code - does it store the top level values as bipf arrays?

The layering makes it a bit easier to work with. The top layer (butt2 msg + content) is so we can easily send it over something like ebt. The bytes of butt2 msg is used to get the key and the bytes of encodedValue is used for signatures.

The important question: how does batch signing work?

You sign the concatenated bytes of the previous N hashes. That works as a signature of those messages. You chunk them into reasonable sizes in order for a debouncing validation to be able to queue up enough messages to use this optimization.

I think you could make a much simpler approach: since hashes preserve the uniqueness of the hashed data, it means signing the hash is as good as signing the data, and signing the hash of the hash (or the hash of data containing the hash) is also good enough. So therefore, every ssb message signs not just the message content but the whole chain of preceding messages. If you changed one bit in the first message the hashes of all subsequent messages would change.

If I understand you correctly, then why didn't we just check the signature of the latest message in classic ssb as well?

@Anders %YYU0/Kpht5Rp/lI1J4PXaNGX+JjREjQTnqyz8WBGsSI=.sha256

To answer my own question. In both cases, random sampling and bulk signatures, some of the individual signatures that you don't verify could be invalid. And they are part of the chain so you accepted some messages that another peer might not. There is a difference though, in the bulk signature case there is always a chain of signatures that correctly signs the whole chain.

@Dominic %//inr6fhPnhKaVTuMsuFfd44GzwPdu8OpAWNhbeGO9Y=.sha256

I think I considered that but didn't do it because like what happens if signatures are invalid? also at that time, signature verification wasn't the bottleneck, it was index building. dropping signature verification then didn't give much benefit. Now that index building is really fast, it's time to reconsider it options like that and batch signing ;)

I bet there is a thread where we discuss that? I've got a funny feeling that we may have played with something like that when we first started making lite clients or something?

Other points, by choosing the signatures randomly, an attacker can't predict which signatures will be checked, so if you have mixed in invalid signatures, then some peers would detect that. And the hashes would still be valid... so you will still have a secure chain. From at least one signature to preceding hashes. You could check individual signatures back to the checkpoints at read.

You could have a deterministic scheme, where a peer would chose what signatures to validate based on something secret but deterministic, so that an attacker couldn't predict what signatures a particular peer would validate. That would also let you gaurantee that it's very likely that all signatures would be verified by someone. Here is where fork/invalid proofs would be useful...

But by far the best argument for this scheme is that it would work with the millionish already existing messages. (how many messages are there these days?)

But either way, neither of these proposals are mutually exclusive. Choosing one does not prevent the other. The batch scheme still checks the main signature, so a peer thats following the old verification style will still validate it.

Okay, that's enough from me. This is your protocol now not mine and I've said my piece so I'll leave it up to you.

Join Scuttlebutt now