Proposal for a Simple-Validation Replication RPC

When an ssb server receives a new message, it verifies its signature. The message may be sent in a different format than that used for signing, this is true for the current json encoding, and it may be true for future compressing encodings. So in order to verify a message, the server needs to know how to encode it in the correct format. In the case of json messages, this is really painful.

I propose adding a new rpc method to sbot that works like createHistoryStream, except it emits objects of the form

{
    crypto: ".sig.ed25519", // the cryptographic primitive of the signature
    signature: [0, 0, 42, 17, ...], // the claimed signature of the raw message
    raw: [0, 42, ...] // the raw bytes that can be given to the signing function
}

The receiving server runs the signing algorithm specified by the crypto field against the raw buffer. If it matches the signature, the message is verified. Decode it, then add signature to the metadata of the decoded object.

This means that servers only need to know how to decode all signing encodings previously used by ssb, but they do not need to know how to convert from transport to signing encoding. The server implementation itself only needs to implement a single message encoding, which it can use for producing its own messages.

This is not intended to replace createHistoryStream, only to allow supplement it.

Drawbacks of this:

• There are usually good reasons for having separate signing and transport encodings, this function ignores them. In the case of the json encoding, that means sending a bunch of whitespace over the wire.
• A server that does not know how to restore a transport encoding from a signature encoding can only relay those messages via the same rpc to other servers.

Advantages:

• allows new server implementations to quickly talk to the main network, fosters experimentation
• reduces the work that each new format imposes on all server implementations
• any implementation can still chose to support old transport encodings for efficiency
• I will never be forced to implement the bizarre json encoding

Of those advantages, the second one alone is enough in my opinion to justify this rpc. The first one will be extremely valuable as well. The fourth one should be completely irrelevant to the decision on whether to add this, but it is true nonetheless, and it makes me happy.

A few more points on potential drawbacks:

• any "serious" implementation should use ebt replication rather than createHistoryStream anyways, so this should not have a lot of impact on overall network performance
• this warrants persisting data in the form it was received in, but iirc the go implementation currently needs to do that anyways (can't find a link though, I might misremember this)
• implementing this should be rather easy, so this will place little burden on future implementations.

CC @Dominic @cryptix @keks @cel @dinosaur

I wanted to look at how much effort this would take to implement. As a starting point, I chose to see how createHistoryStream is implemented and where it gets used. After 15 minutes of trying to find its definition, I've given up. I'm not even sure which module to look in. Started in 'scuttlebot', then meandered to 'secure-scuttlebutt', which seems to load an object called db that refers createHistoryStream by name from 'level-sublevel/bytewise', which is a function that takes 'level' as an argument, at which point I decided to ask instead.

• Where is getHistoryStream defined?
• How does one add a new rpc to #sbot?

@Aljoscha

The createHistoryStream function is defined in secure-scuttlebutt/indexes/clock.js. It is added to a db object in secure-scuttlebutt/index.js along with other functions. In scuttlebot/index.js, one of those db objects is returned from the secure-scuttlebutt/create.js constructor into a variable, ssb. The init function in scuttlebot/index.js returns an object with methods, including passing through a few ssb methods, some wrapped with a validation function. Those methods become top-level RPC methods for that scuttlebot instance. Also in that file there is a permissions object adjacent to the init function, which allows the anonymous permission class (i.e. anyone) to call the createHistoryStream method.

You can define a #sbot RPC method by editing scuttlebot/index.js, or by adding it in a plugin. The scuttlebot module (with main function in scuttlebot/index.js) returns a constructor function createSbot, which has a use function which you can call with a plugin object to make the constructor load that plugin object. (The use function is defined in secret-stack/api.js). The scuttlebot (sbot) command is scuttlebot/bin.js and you can see the plugins loaded there with all the .use() functions called on the createSbot function. Scuttlebot plugins are loaded similarly in patchwork/server-process.js, patchbay/background-process.js, talenet/sbot.js, and scuttle-shell/server.js.

There is also a way to load plugins without editing the JS files that call createSbot. scuttlebot/bin.js and scuttle-shell/server.js will load the plugin scuttlebot/plugins/plugins.js, which then loads additional "userland" plugins, from ~/.ssb/node_modules by name according to the plugins scuttlebot config object. scuttle-shell also loads additional plugins from paths specified in a scuttleshell.json file.

By following the code for what gets loaded by the createSbot.use function, you can see how plugins are structured. A plugin is an object with an init function that gets called with scuttlebot instance and ssb-config instance, and returns an object with RPC methods (which may also get called locally by scuttlebot or other plugins). As well as the init function, the plugin has a manifest property that defines the muxrpc type of what methods the plugin defines (as returned by the init function. There can also be a permissions object to help specify which peers can call the plugin's methods. The plugin object also has a name property, which is used to namespace the RPC methods. (so gossip.connect is the connect function from the gossip plugin). If no name is given, the plugin's methods are loaded as top-level methods - but userland plugins are not allowed to do that. The plugin object also has a version property which is required for userland plugins, but I have not seen it used anywhere.

Wow @cel, thank you so much!

Any recommendations/opinions on whether createSimpleValidationStream (also open for better naming suggestions btw) should be its own tiny plugin, part of sbot, or part of secure-scuttlebutt?

@Aljoscha you're welcome. I think I would suggest it be in its own plugin.