plugins

The main goal of plugins is to make it easy to extend secure-scuttlebutt, i.e. lower barriers to developers. But the flip side of the coin, is that there is not much reason for developers to developer if they can't get users to install their plugins. So the second goal, equally important as the first, is to enable users to feel confidant about installing a plugin. Because, lets face it: any plugin system is a security hole. Therefore it is essential that we design a carefully considered security hole that is easy to understand, so we understand what sort of hole we are opening, and what may pass through it.

Also, there is also one specific thing we already know: plugins need to be separate processes, because many members of the ssb community can't be persuaded to be happy coding in javascript (I accept this, even if I don't fully understand why), and while reimplementing all of ssb in another language is possible, it's a lot of work to get something usable. If plugins could be a separate process, it would break up the process of reimplementing ssb into many small successes and useful contributions, instead of just one big one. It's more incremental! (see upgradability principle)

I think a plugin design should provide the abilities to do the kind of things people are already doing (example: creating back end services for their front ends, extending the database in certain ways, etc). The features a plugin system supports should answer to already known use cases, or well described plausible use cases that someone actually wants to build. I'm not interested in supporting features described abstractly that arn't accompanied with a plausible sounding description of a concrete application that community members would want to run.

Okay, with that all said, here is what I propose.

plugins run as child processes of sbot, and communicate over stdout.

sbot starts any configured plugins when sbot starts, or when the user requests another plugin to run. the user can also stop and disable a plugin at any time. If a plugin crashes, it is restarted, after a short timeout, but any requests currently underway to that plugin return an error. The calling plugin needs to handle that error, although crashing plugins usually mean a broken application, although sometimes retrying would work. However, the decision of which way to handle this is left up to the caller. I think this would be more than enough to implement any of the currently implemented applications.

If a plugin provides methods to clients or other plugins it needs a name. (there are some current javascript sbot plugins that do not provide an api - and a concrete example would be a plugin that calls the friends plugin, then calls the replication plugin or gossip plugin to tell them who to replicate or connect to (as discussed in the untangling sbot thread)

You could have a design where a plugin could just call any plugin, and any calls to the plugin are directed based on the name. But, I think it in better service of the user (2nd goal) if a plugin declares what methods it provides, and what it will call on other plugins too. Maybe there are plugins that want to be able to call methods dynamically - there can be a way to express that, but it doesn't need to be the default behaviour. Also, a manifest doesn't preclude a plugin from using it's own protocol: just declare a single duplex stream, then speak a custom protocol over that.

If the plugin speaks on stdio to sbot, then it only has a single connection. It doesn't even need bindings to networking (this is definitely preferred, preventing plugins from making arbitrary network connections protects the user from surveillance). Lets say that a plugin can call the methods provided by other plugins (provided they declared their intention to do so in a manifest). Those connections are just routed to the plugins. To call an api on a remote peer, a plugin would call a method to get list of currently connected peers, (may be a stream of notifications about this) and then call a special method with the id of the peer they want to call a method on, the api could be: remoteAsync(peer_id, path, args, cb). remoteAsync(peer_id, 'get', {msg_id}, cb). Or something similar.
However, I think that current applications do not require this feature, so it can be left out for now.

Once we have the basic metaphore of constrained communication between peers, if those plugins are then run inside some sort of sandbox/container then a high degree of security can be gained.
My goal here is to have enough security so that plugins can simply be distributed via blobs, and installed and even updated at a click. But, at this current point in time, that isn't needed. Currently plugins get a lot more trust than that, but given the small community of people building them, it's not currently a problem - but if we are successful, I think this will become a problem ;)

somewhat forked from %eOgrGRI...

Happy to read this. Much more in line with what we discussed previously and looking forward to prototyping a stdio plugin next week!

where previous is %oxtXLx0... and %a0K84ZS...

@Dominic I think we actually agree on a lot of points, or at the very least on the goals. My main ask is simply to not rush this.

My thread on plugins is mostly me thinking out aloud while exploring the design space. I still don't feel like I have a full grasp on a lot of the details. My mode of operation is to figure out the most general way of doing things, and then use that as a foundation for making decisions - which will certainly involve restricting the scope. So I guess I'm asking you to humor my explorations, knowing that I am open to reduce the ambition of the current drafts.

This approach may lead to the same result - sandboxed, local plugins that declaratively specify their api with constraints enforced by the server - but I want to fully understand how to reach those conclusions, and what the alternatives are. You clearly thought about these things longer and much deeper than me, cryptix and keks did, but we can't look into your head.

To give a vague, mathematical analogy: I'm trying to find linearly independent vectors that span our whole problem space, before settling on a set that only spans part of the problem space. Because who knows - maybe there's a really sweet spot outside of what a more constrained approach can find.

@noffle Yes definitely - Although because it's a separate process there are more copies, so there might be a performance disadvantage, but we won't know how significant that is until we try. ssb-query can now accept indexes provided externally, but it would be possible to just pass it the paths that you'd want to have indexed. (would this be enough for your needs?)

the paths you want indexed

is the sequence of fields that will be looked at. For instance .content.type, or .content.custom.struct.zzz if you have lots of those. In ssb-query those are these by default.