problem: secure-scuttlebutt has encrypted messages, and lots of indexes. This means that indexes which need to index the contents of those messages need to decrypt that.
in secure-scuttlebutt code, only the links index decrypts anything, but in patchwork, both ssb-backlinks and ssb-query decrypt things on their own. That means for patchwork to build indexes, it spends 3x as much time decrypting as it needs to.

So, having the database respond only with the cyphertext seems like a bad thing, although on the other hand, it does mean that we don't need to worry about the plaintext leaking out. if you want the plaintext, decrypt it! If ssb responded with the decrypted plaintext we (I mean I) would have to think very carefully about what apis are possibly returning plain text to who.

But! it could be worthwhile, for example: if all the indexes worked with encrypted messages: you could privately follow or block someone. You could like something, but keep it to your self (as a bookmark) or to the author of that message only. To implement these features, the frontend would just have to have an option to post those message types encrypted. This will become especially interesting with private groups.

If ssb were to use decrypted messages internally, that would mean we'd have to re-encrypt messages before sending them to remote peers. (or rather, to keep the cyphertext about, and replace the content with it before stringifying) I dislike this approach because all I have to do to leak your private messages is miss something, were as currently with everything private I am not worried about it.

One approach, would be to have a serializer that noticed the message was encrypted, and replaced content before sending. So just never send decrypted messages over the network. internal plugins would have access to decrypted messages, but not peers.

Another approach would be to handle the main replication functions (createHistoryStream - used by legacy replication, getAtSequence used by ebt, get used by ooo) returning decrypted messages would surprise most clients also. probably it should always be opt-in you'd have to set a private:true option, otherwise you get sent the encrypted message.

hmm...

can you not do something more naive:

• have flumeview plugins loaded in-order so that a new module, say ssb-predecrypted when plugged in first gets passed messages first.
• then when a plugin comes across an encryped message, you send a request to ssb-decrypted to handle it, which has been building a plain text cache

e.g.

server
  .use(require('ssb-predecrypted'))
  .use(require('ssb-backlinks'))

//// in ssb-backlinks

  if (typeof msg.value.content === 'string') {
    msg.value.content = sbot.predecrypted.getContent(msg.key)
  }

you could even pass every message over and put that boolean logic inside ssb-predecrypted

@mixmix interesting suggestion. But apart from performance, I don't think it addresses my other concerns:

• avoid updating every plugin/view, but get to use privacy layers on any message type.
• make sure the views didn't leak private messages via client requests

Okay, I took a stab at this... convieniently, there was a util.formatStream that everything used to handle wether you had keys or values in the output. This is pretty much a secure-scuttlebutt standard, so I just added a private option to it. If private is set to true, then it will give output that is private, otherwise false. So by default, all the current uses will work the same. if you want private data, you must explicitly set private: true

If you request a message by it's sequence (used internally by indexes) you'll get the decrypted message, so indexes will index private data by default.

Pretty sure none of the other plugins return data over the network, only local stuff.

I'm not gonna merge this till after I've carefully checked all the plugins.
it's up at the v18 secure-scuttlebutt branch. I don't recommend running it yet, until after a careful review!

But I have got all the secure-scuttlebutt and scuttlebot tests passing, and patchless still works (after removing ssb-private).

@keks that sounds somewhat like what I have in the v18 branch. if a message could be decrypted, it's content property is set to the plaintext, and a cyphertext: ... and private: true properties are added. Then on certain apis, the messages are "reboxed" which just means the cyphertext is copied back to content. It would be possible rebox before passing to particular plugins, but I wouldn't really consider that security unless they actually ran in a sandbox of some kind.

Interesting approach. This will clean up a lot of code in external plugins!

I think we need to be really explicit about which functions are exposed externally in the documentation (https://scuttlebot.io/apis/scuttlebot/ssb.html). In general I think the API docs could really use an overhaul. Which functions are old/deprecated. Anyone needs to be updated to flume? Maybe something for @jolyons developer docs grant?

@arj Related question: %YEPJB3p...

Given the keys,values pattern in all the internal methods, a simple way would be to expose a sbot.format({keys,value,private}) method, and strongly recommend that plugins exposing streams use it. It would then enforce that no private data is returned, unless explicitly asked for (private=true) then, using muxrpc's auth stuff, it can just check that {private:true} is never passed to sbot unless it's an authorized connection.

@keks I don't think this is an issue with the currently implemented plugins, mostly they don't call each other, and i'm fairly sure that the only plugins that communicate across the network are ooo, ebt, ssb-viewer and ssb-irc (hadn't though of ssb-irc till just now, which is why I plan to go through and review) viewer and irc arn't really that dangerous since they run from pubs and so if as long as the pub isn't a recipient of a private message then it's okay.

My immediate goal is just to make the current system work better and be easy to check that plugins are safe.

There are some interesting research I've heard vaguely about, where they like, trace the execution of a process, and can tell if it's possible for a private data to be in an output. (basically, they trace the causality flowing out of private data, if something is caused by private data, don't allow that to leak)... maybe if we can attract people researching this sort of stuff to ssb?

@keks yeah, definitely. I think it probably be easiest to do it at a bytecode level. For example in wasm. (when I heard this idea, it was being applied to machine code). Probably needs to be statically compiled code, not jit or interpreted.

Okay, took another swipe at this: pr for decrypted indexes

I am gonna sit on this a little while but I think it's probably okay to merge.

Okay, took another swipe at this: pr for decrypted indexes

I am gonna sit on this a little while but I think it's probably okay to merge.

Awesome, keen to start using this in patchwork

We'll need to update ssb-backlinks and a couple of others.