project: tests for ssb implementations

This is my idea to help get people on board working on ssb implementations.

The main idea is to present structured tasks that guide people towards expertise regards ssb.

the essential things

• incremental: a little bit of work still takes us closer.
• unifying: helps pull the various implementations together.
• neutrality: we want to avoid priveleging a particular implementation, (although out of necessity, the node.js implementation may be treated as a reference implementation, but we should move away from that)

My main idea, is test cases that can be used across implementations.
Tests should be written in a neural, language that is available everywhere.
I started some in bash

Tasks can be writing tests, or improving implementations so that they pass tests.
But implementing features that do not have test coverage does not fit into this project. There must already be a test for work to come under this project.
Even if you pass one more test, that is progress. And if you write a test that one implementation can pass, that is progress.

(we want to make a feeling of progress as easy as possible! working on new features is hard, because you either succeed or not, and you need a lot of knowledge before you can usefully contribute. So we want to frame contribution in a way that doing something is easy)

Implementors, what do you think? @cryptix @keks @aljoscha @cel

The current tests check that the output has the right hashes - so every bit is the same. This is appropiate for some things, but others should probably be relaxed a bit. Currently the tests also cover the database api - which are not really technically part of the protocol. However, there are probably some aspects of this that we'd want to have shared across implementations, or at least some implementations.

Totally open to discussions as to what features we want to have shared, and not to expect all implementations to implement everything - but to at least have these tests as a guide to tell us how compatible they are.

This is a good idea. I did a test suite for shs a year ago, but that one dynamically performs hanshakes (and is written in js) rather than replaying pre-recorded data.

Some suggestions for test cases on the transport-layer:

• shs
• box-stream
• box-stream with secret negotiation via shs
• packet-stream and packet-stream-codec

To tie it all together: A really overengineered calculator: Test suite provides the server, which does a handshake, encrypts via box-stream, then takes muxrpc (both req/res for simple calculation and duplexes for larger calculations) over packet-stream and sends back the results. The implementer writes a client that sends some set of computations to the server and must write the received responses to stdout.

This is something that would have been immensely valuable to me when I worked on the Rust implementations of the transport layer protocols.

Also very much in favor of this!

For #go-ssb we already have basic interop tests. They run when you use go test -tags interop_nodejs

• secrethandshake
• boxstream
• go-muxrpc
• muxrpc codec

I wanted to add this flavor of tests to our #golang sbot but couldn't figure out a good setup yet(*) yet but I didn't so far because I wanted to make feature progress first and felt like there should be something I can reuse. I'm not sure if @arj CI-work maybe already has something for this that could easily be ported.

Coverage-wise I'd also argue to do the network layer first. There is a lot of stuff in there. The ssb-validates suite would also be very handy if it could be used as a suite. I'd say lets go for basic feed replication, blobs and maybe invite mechanics.

re setup: Ideally I want a lot of scenarios. (un)followed, blocked, forked feeds... Do I fill the js node first? How do I signal completion without timer madness (could add muxprc methods for this).

ps: yikes.. I just stombled over the old shs1-testsuite branch on secrethandshake.. I thouhght I'd completed this already.

We don't really need to make the cli authorative, whats more important is to make the RPC the authoritive thing. However, the CLI is just a muxrpc client - and there are already more than one cli, node.js's and @cel's sbotc.

So we could run the tests using one server and a different client. Implementing a client covers muxrpc, shs, box-stream, so you are making some good headway to get to that point.

I think the next step is getting at least something from the various clients passing the test. Like, if they can just answer whoami that's something.

Another idea is a project I was made aware of yesterday called zenroom where the crypto could be specified in language neutral wm and be shared between the different implementations. This does not mean that we don't need tests, just that the really tricky crypto stuff can be written once.