Why does my brain keep doing this? I promise, I didn't want to think through this stuff... But I might as well write it down now. CC @Dominic @dinosaur @Piet @cryptix @keks @cel

Thoughts on ipc-based plugins

These thoughts are guided by the desire to

• move as little work into the server as possible
• move as little decision power into the server as posibble
• make the server know as little about the plugins as possible
• allow sandboxing
• provide a standard mechanism to facilitate plugin-to-plugin communication

Without the last point, all of this becomes nearly trivial, we could mostly treat plugins how we currently treat clients. But the last point is also the only one in conflict with the other goals, each proposal will inherently be about making tradeoffs between them.

This discussion will also involve the distinction between server-to-server and server-to-client, and where plugin-to-plugin fits into it. Aaand there are encoding considerations about sending short integers as rpc identifiers rather than utf8 strings.

And a final warning: I don't care at all about manifest files and what they claim regarding permissions. If you talk to another process, you might send or receive any messages (or bytes actually), no matter what some sort of manifest file says. And as for permissions: Either the other endpoints does what you asked for, or it does not. Again no ties to any sort of magical file.

Plugin-to-plugin communication

Since this is what all boils down to, I might as well start here. The basic idea is that plugins expand the range of rpcs an ssb server is capable of handling. There are a few rpcs in the core, and then it might delegate other rpcs to plugins.

Addressing plugins

I basically see two reasonable approaches for making sure that an rpc reaches the correct plugin: Anonymous plugins (or globally scoped plugins) and named plugins (or namespaced plugins).

In the case of anonymous plugins, the server would receive an rpc (which contains a well-known identifier), then check whether one of its plugins is able to handle an rpc of that identifier. In the case of named plugins, each plugin comes with a well-known identifier, and rpcs specify the plugin to which they are addressed. The server just delegates based on the identifier.

At first glance, there seems to be a big advantage of the anonymous approach: An endpoint does not need to know whether a certain rpc is provided by the core, or a plugin (or which one). But on closer examination, this does not really hold. Any rpc mechanism only works if the communicating parties agree on the same, well-known identifiers for the rpcs. But if such an agreement is necessary, then they can just as well agree on the well-known name of the plugin that implements the rpc.
But named plugins actually reduce the burden of coordinating to find well-known names. They are effectively namespaces for rpc calls, as long as two plugins use different names, they can identify their rpcs however they like, including short integer identifiers.
Anonymous plugins also need to handle partial naming conflicts. It would be unsatisfactory to reject a whole plugin just because it uses the same rpc method name as some other plugin, but everything else (e.g. first come, first served) could lead to problems where the peer expects a different plugin to handle its rpc calls.

While I described the named plugin mechanism as sending the plugin name with each rpc, that's just a conceptual model. In practice, we will want to use bpmux/muxrpc channels to bundle all communication to a specific plugin. And those automatically use short identifiers rather than having to resend the well-known name each time.

So overall, namespaced plugins are easier to coordinate between different developers, are more efficient, and leave complete freedom on how the plugin interprets what kind of rpc a message represents. In fact, the server does not care about the wire protocol of the plugin at all, the plugin could use protobufs to communicate if it wanted to.

continued in next post...

...continued from previous post

Connecting plugins

Plugins need to be able to interact, for that we need to reimplement dbus provide a few basic facilities. Each plugin should be able to:

• query which plugins are currently connected (by well-known name)
• be notified whenever a plugin connects/disconnects (by well-known name)
• create a bidirectional communication channel to another plugin (you guessed it, by well-known name)

The last one is the big difference between name-spaced plugins and global plugins, and I think it is the cleaner solution. Not only does this allow plugin-to-plugin communication to use any sort of rpc protocol they want (in particular, a fully-fledged rpc protocol might be overkill), but it also allows direct connections. The simplest implementation for the server is to hand out a logical channel to the plugins, which first sends data to the server and the server then reroutes it to the other plugin. But a more advanced implementation could instead mediate creation of a direct channel between the two plugins. On the same machine, that could mean creating a unix domain socket and would save a few memcpys and syscalls. But for plugins living on different machines then their server, this would mean a direct tcp connection rather than (literally) rerouting everything trough the server. This is another advantage we just could not get with anonymous plugins.

There are a some details to work out for this, and there are some considerations regarding the capabilities required for querying connected plugins and establishing connections, but overall, this is surprisingly simple.

Plugin startup

If the server has to start plugins, it needs to know a bunch of things about them, and it also needs to decide when to start them. What is a design where the server doesn't need to care? It can expose well-known sockets (unix domain, tcp, whatever transport you want to use) where processes can register themselves as plugins. "Registration" sounds fancier than this needs to be, at the bare minimum the plugin would just say "Hi, I exist, and I go by the name plugin_foo." and the server would respond "Cool, here's a bidirectional communication channel we can use for all remaining stuff", while notifying subscribers that the plugin connected.

We'd probably want to perform a secret-handshake first, so that the connecting process (which may or may not live on the same machine) proves that it has the capabilities of a certain identity (i.e. the secret key corresponding to a feed id), which may or may not be the same as the server's identity itself. No need to encrypt the connection if it is on the same machine, but it still makes sense to identify via shs.

Does this sound familiar? It is how clients currently work, except for the part where a plugin specifies a name.

With that done, there's a dedicated, bidirectional communication channel between the server and the plugin. At the very least, the plugin should be able to call all the rpcs that a client can call. But in addition to that, there's the plugin-to-plugin communication as described above, facilitated by the server.

Unifying clients and plugins

Wait, what? Well, with this approach, plugins are basically a generalization of clients. A client is simply a plugin which not offer rpcs to other plugins. That's why it does not need a name. So all we need to do is to allow nameless plugins which do not show up in the plugin query/notification mechanism. To the server, there's no distinction between clients and plugins anymore, which will simplify a lot of things.

Remaining thoughts

(Jotted down in a hurry)

I brushed over authentication details: When opening a channel between two plugins, they should each know the public key of the peer (and the server needs to enforce that they have the corresponding private key).

Also, it would be possible to take a much more opiniated approach, forcing all plugin comunication to happen via bpmux, etc. This might help with consistency, but overall plugins still need to come to agreements, so they might as well agree on the encoding of the communication mechanism. In practice, prettty much everything will likely default to bpmux anyways, and those who don't - well, they'll have good reasons and be happy that we didn't enforce it.

A few more thoughts:

• this approach would greatly simplify this invocation for modular clients, to implement it, you'd simply need plugins (for the communication between clients) with a UI (for the communication with the user).
• each plugin should be able to decide who gets notified of its presence, for example based on the following modes:
  • explicit permission: Whenever a new plugin wants to query information about you, you are sent an rpc with the name and pubkey and get to decide whether that plugin is allowed to learn of your existence. This is the most general mode, but for efficiency (remember that plugins might live on a different machine) it makes sense to provide a few more:
  • always: Everything is allowed to query information about you
  • trusted: Everything with the same pubkey as yourself or the server is allowed to query information about you
  • server: Everything with the same pubkey as the server is allowed to query information about you (I think this is called "local" in current muxrpc/manifest parlance?)
  • same: Everything with the same pubkey as yourself is allowed to query information about you
  • never: Nothing is allowed to query information about you.

More vague addendums:

• If the server is to mediate direct connections between plugins, it needs to know what kind of direct connection a plugin supports. Since a plugin might live on another machine, the server needs to know, whether that server has e.g. a public ip address, or whether hole punching is necessary. This can be solved in a very simple fashion: Wen a plugin registers at the server, it specifies a set of multiserver addresses that can be used for setting up direct connections. This way, it would even be possible in a setting where two plugins share a machine but the server lives on a different machine for the server to mediate a unix domain socket connection between the two plugins.
• When two servers connect, they'd then query for each other's plugins, and then establishing connections between those plugins based on the result of the query. As an optimization, it should be possible to send messages saying "If you have the foo-plugin, please use one of the following multiserver addresses to establish a connection to my bar-plugin.". That saves a roundtrip before the plugins can talk to each other.
• Just as a clarification: In a setting like that described above, if plugins live on the same machine as their server, instead of setting up a new physical connection, the connection between remote plugins would be multiplexed via bpmux over the original connection between the servers.
• With this plugin architecture, pretty much all functionality of an ssb server except for plugin setup itself could be provided by plugins. This would reduce the pressure on us to figure out a set of "core rpcs" everyone has to support, instead there'd be a few orthogonal core plugins. Since servers exchange information about which plugins are present, this would serve as a kind of feature announcement. This avoids roundtrips just for finding out that the other endpoint does not support ebt, because you'd only be able to send rpcs to the ebt plugin if it was available. If a server finds that the peer does not support any plugins for them to replicate with each other, they can simply drop the connection. As a consequence of this, we would not need to specify mandatory core plugins, this could instead evolve dynamically.

• Versioned plugins: Instead of just exchanging plugin names, exchange plugin names plus a version varint. All new versions must be backwards compatible, so if a plugin wants to talk to to ("foo", 4) a connection to ("foo", 7) is ok, but ("foo", 3) is insufficient. For breaking changes, use a new plugin name.
• This whole plugin architecture (bpmux + multiserver + plugin management rpcs) is completely independent from ssb. Ssb would just be a loose collection of plugins that utilize the plugin architecture to exchange hsdt messages.
• A consequence of this: Plugins can be application agnostic. You could write e.g. a gossiping plugin for creating overlay networks that could be used by ssb, but also by any other sort of application built upon the plugin platform.

I've been mulling this over a lot, and I feel like my thoughts are slowly converging on a coherent proposal, but that'll take some more time to write up. For now, just more notes to self.

It's hard to limit the scope of this plugin platform, there's the temptation to stay completely unopinionated yet provide all the useful features. Libp2p is a cautionary example in that regard. Reading the website, it looks like the silver bullet that solves all problems. Until you dive into the spec and see all the open problems, partially or underspecified aspects, the bloat (why is there a pubsub service on this protocol layer, isn't this the protocol that pubsub should be built upon?), things marked as TODO, etc.

A realistic, actionable proposal will need to make a few opinionated choices about which use-cases are out of scope, to keep things manageable. I think direct connections between plugins won't actually make it, as these come with a lot of baggage. Different pubkeys per plugin might be ok, I think, but I'm not sure yet (the more restrictive approach would be that each plugin must have the same pubkey as its corresponding server).

And finally there's the question of how much the plugin-server should be aware of connecting to other plugin-servers. There's a few approaches here:

• Don't specify this at all, resulting in a plugin architecture not tied to p2p systems in any way - p2p functionality would be given by plugins.
• Transparently handle p2p connections, maybe giving plugins the ability to adjust parameters. This is the easiest one for plugins, but the most restrictive/opinionated way to go.
• Give an opaque set of primitives for connection establishment and address sharing to plugins, but leave it to them how they use those

I'm currently leaning towards the first or third option, but haven't made up my mind. Either of those three would be suitable for ssb, it's just a question of where to place the inherent complexity, and how much complexity to shave off by reducing flexibility.

This is not a proposal for ssb. This is just me synthesizing a few thoughts on plugin-based architectures in general, while completely avoiding the open questions, namely the notion of (un)trusted identities, and handling p2p connections.

Changes to this spec are published on github.

Min-Bus

A somewhat minimal system for coordinating inter-process communication.

Abstractions

Min-Bus is a system that allows different processes - called plugins - to discover and talk to each other. To do so, it provides a server, where plugins can connect to. When connecting to the server, a plugin must specify a name (arbitrary byte string not longer than 255 bytes), and a version (unsigned 16 bit integer). Connecting succeeds if no other plugin with the same name is currently connected. Versions signify non-breaking changes to the API exposed by a plugin. If a plugin performs breaking changes to its API, it needs to change its name.

Once the connection is established, the plugin can perform the following actions:

• query the currently connected plugins (name + version)
  • this opens a stream where the server first sends the list of currently connected plugins, and then sends a notification whenever a plugin connects or disconnects
  • the plugin can cancel the stream at any time
• be notified of other plugins that what to connect to it. It can either reject the connection, or allow it.
• ask to be connected to another plugin (by name). If the other plugin accepts, both of them are given
  • a bidirectional, byte-oriented communication channel to the other plugin
    • with an explicit, credit-based backpressure mechanism
    • with heartbeats
    • either plugin is allowed to close/cancel it
    • plugins can use any protocol they like to communicate
• disconnect

Implementation

A server implementation is free to choose any mechanism for plugins to connect to, as long as there is a bpmux implementation that runs over that mechanism. For example, the server could listen for tcp connections on a specific port, or it could use unix domain sockets. A server is allowed to expose any number of such mechanisms (e.g. both a local unix domain socket and a tcp socket with a public address), all plugins can talk to each other regardless of how they connected to the server. For public-facing ports, it is recommended to enforce authentication and to encrypt all data.

After connection establishment, all communication happens via bpmux. The server should should always supply the plugin with enough credit so that it can perform queries. The server should also respond to any heartbeat pings. The server may send heartbeat pings to a plugin and disconnect it if it does not react after a well-documented, implementation specific time.

Registration Message

Initially, the server does nothing but wait for the plugin to send its registration message. This is a message that consists of an unsigned 8 bit integer, followed by that many bytes as the name of the plugin, followed by a uint16 version of the plugin. If there is already a plugin of that name, the server simply terminates the connection. Else it notifies all subscribed plugins that the new plugin has been registered and goes into the connected state.

After sending the registration message, a plugin can send any of the following to the server:

Disconnection Message

The plugin sends a message with a zero-length payload. This is not strictly necessary, the plugin can instead directly terminate the underlying connection. But in case of connection types where this can not be directly detected by the server, it is polite to send this message prior to disconnecting.

Upon receiving this message, the server does all the cleanup necessary for disconnected plugins (see section "Server Behavior Upon Plugin Disconnection")

Connection Query

The plugin sends a request with a zero-length payload. The server sends a response containing the repeated <length_uint8><name><version_uint16> tuples of all currently connected plugins, except for the requesting plugin itself.

Connection Stream

The plugin opens a stream with a zero-length payload. The server immediately sends the set of currently connected plugins (except for the plugin itself) as a single message to the stream, containing repeated <length_uint8><name><version_uint16> tuples. Afterwards, whenever a plugin connects, the server sends a message containing the byte 0000_0000 followed by the <length_uint8><name><version_uint16> of the plugin to the stream. Whenever a plugin disconnects, the server sends a message containing the byte 0000_0001 followed by the <length_uint8><name><version_uint16> of the plugin to the stream,.
If a plugin opens a connection stream while it still has another one open, the server must immediately cancel the new one (with a zero-length payload).

continued in next post...

...continuation from previous post

Connect to Plugin

For plugin a to establish a connection to plugin b, plugin a opens a duplex with a payload consisting of the byte 0000_0000 followed by <length_uint8><name> of plugin b. If a has already established a connection to b that hasn't been closed/canceled in both direction, the server must directly cancel and close the duplex with a zero-length payload. Else, the server must allocate enough memory for two 64 bit integers and two buffers of bytes for caching data, each of size at least one byte. If it fails to do so, it must both close and cancel the duplex with the byte 0000_0000 as the payload. If memory allocation succeeded, the server then opens a duplex to plugin b, with a payload consisting of the byte 0000_0000 followed by the <length_uint8><name><version_uint16> of plugin a.

All pings sent by a get relayed along that channel. All credit given by a is buffered into one of the 64 bit integer the server allocated. Likewise, pings by b are forwarded and credit given by b is buffered as well.

Plugin b can deny the connection by closing and cancelling the duplex with a zero-length payload. In that case, the server must close and cancel the duplex with a with the byte 0000_0001 as the payload. Plugin b can accept the connection by sending a zero-length message down the duplex. In that case, the server must send a zero-length message down the duplex it shared with a. It then sends all buffered credit to a and b respectively. The logical connection between a and b is now established, consisting of the two duplexes.

After the connection has been established, the server forwards all messages, heartbeat pings, heartbeat pongs, and credit along the duplexes. To the plugins, this looks like a direct connection between them. There is a situation however where the server may choose to not directly relay all credit. Imagine a setting where there is a very fast connection between a and the server, but a very slow connection between the server and b. Furthermore assume that b gives a lot of credit to a. a sends a lot of data, but the server can not relay it fast enough to b, so it has to buffer data. In this setting, b would effectively determine how much data the server has to buffer. To prevent this, the server can simply put an upper limit to the credit it hands to a, and store any excess credit given by b in one of the 64 bit integers. When more data could be sent from the server to b, it can then send more credit to a, removing it from the 64 bit integer. The server should use a scheme that is more efficient than sending credit whenever data could be sent, as that might result in a lot of credit data being transmitted.

If one of the plugins cancels/closes parts of the connection, the server prepends a byte 0000_0000 to the payload before relaying it.

Server Behavior Upon Plugin Disconnection

When plugin foo sends a disconnection message, or the connection to the server is closed or errors, the server must perform the following cleanup work (only once per plugin):

• notify all connection streams
• for any connections between foo and other plugins, cancel and close the duplex between server and the remaining plugin, both with the byte 0000_0001 as the payload.

Handling Unexpected Messages

If the plugin sends any data to the server that does not fit into one of the flows described above, the server must do the following:

• if it is a message, ignore it
• if it is a request, cancel it with a zero-length payload
• if it is a sink, cancel it with a zero-length payload
• if it is a stream, close it with a zero-length payload
• if it is a duplex, cancel and close it with a zero-length payload

This allows backwards-compatible protocol extensions. Clients can use these to check whether some feature that might be added to min-bus is unsupported by the server, without disrupting the connection.

To get from min-bus to a plugin architecture for ssb, there's a few things to tackle. First of all, there are some engieering concerns/optimizations for reducing latencies and round-trip times. But more importantly, there's the question of how much the system should be concerned with p2p stuff.

The absolute minimal extension would be to add identites to min-bus (I guess that would yield the "minimal identity bus", aka mini-bus...). The server has a public key, each plugin connects with a key, there's a handshake to confirm them. Keys are transmitted in addition to plugin names. You could call it a day and have a plugin system that handles identities, suitable for ssb. But that's not enough, there are merits in making the whole framework more aware of the p2p/distributed nature.

Fundamentally, in any p2p setting, there's the notion of trusted (local) computation, and untrusted computation done by peers. Simply adding and enforcing identities to plugins does not capture that at all. Instead, the cleanest solution I could come up with, is to distinguish between two kinds of connections: plugins, and peers. Plugins are the trusted building blocks of the application, just as in min-bus. Peers are not necessarily trusted, but expose (not necessarily the full set of) their plugins to each other.

To implement this, each server has a keypair. When a process connects to the server, they perform a handshake (e.g. shs). If the process uses the same keypair as the server, it connects as a plugin, else it connects as a peer. In addition to the features of min-bus, plugins can also query and subscribe to the connection/disconnection of peers. Peer information consists of the public key of the peer and the names and versions of the plugins it exposes. Plugins can establish connection to a peer's plugins similarly to how they can connect to other plugins.

That's the high level overview of the kind of system I think would be suitable for ssb. Discovery of other peers would be done by plugins (or simply out-of-band). There's one last problem I have with this approach: Any reasonably sophisticated gossip protocol should not conflate gossiping (exchange of peer network addresses) and application-level logic (such as synchronizing feeds in the case of ssb). Usually there is a larger number of connections for gossiping purposes than for application logic. So it should be possible for two servers to connect to each other, but not yet announce their presence to all plugins, but just to a subset of them. Maybe those plugins would want to delegate the connection to some other plugins, but still not open up the full application-layer connection.

I'm unsure about how to handle this, there's lots of possible ways to go, interpolating between ignoring the issue or taking a lot of control. But once that's figured out, I think I can finally a proper writeup/proposal for a plugin-based architecture for ssb, including some engineering/optimization work.

I should probably clarify that the last problem is something of an optimization rather than a real conceptual limitation. It is totally fine for plugins to connect to other processes on other machines that happen to also be plugins under the same architecture, to negotiate connection details, and then tell their servers to connect to each other (a necessary capability of plugins I forgot to include in the previous post). But creating a new connection between the servers incurs a bunch of overhead (connection setup, handshakes, giving credit, querying plugins). Ideally, the servers could reuse whatever connection the (gossip) plugins already had. This would mean adding some complexity to the architecture that is not inherent to the problem space itself =(

Here is another fun nuance to think about regarding min-bus. Should there be a difference between a connection between plugins a and b and between b and a? Under the current draft, these are indeed different, and both can coexist at the same time. It is difficult for plugins to ensure that there is only one connection between them, because of race conditions where both might simultaneously open a connection to the other one. It's possible to work around that, but both plugins need to cooperate. And to at least one of the plugins, it would look like its connection attempt failed. So here are two alternative approaches: An asymmetric one, and a symmetric one.

The asymmetric approach is done through a tiny yet fundamental change: When plugin a wants to connect to plugin b, the server never tells b about the identity of a. Under this model, the receiving side of a plugin is fully passive and must treat all incoming connections the same way. This actually simplifies plugin implementation, as there are fewer decisions to make. A plugin just offers some services to the outside world (not caring about who consumes them), and it can anonymously consume the services offered by other plugins. Proactive and reactive behavior are neatly separated. There's no need to think about consolidating connections a -> b and b -> a, because it's impossible for the plugins to detect anyways.

For the alternative, symmetric approach, the server takes an active role in race conditions where both plugins want to start a connection to the other one concurrently. Suppose a tells the server that it wants to connect to b. The server opens the appropriate duplex to b, but then the server receives b's wish for a connection to a, which b has sent before the server contacted it about a's connection establishment attempt. Under the current min-bus draft, the server does not care and simply relays b's connection attempt. But instead, the server tells a that its connection attempt succeeded, and it also tells b the same. Now,b still got that connection attempt from a, so what happens to that? It was received by b after b sent out its own connection attempt, so b has all the necessary knowledge to just silently discard it. The actual plugin code of b never gets notified of a's connection attempt. There are some more details regarding credit and heartbeats, but overall that's how the symmetric approach would work.

With the symmetric approach, the server must only allow a single connection between two plugins at a time, no matter which one initiated it. When a race condition occurs, each of the plugins thinks its own connection attempt succeeded. So in general, plugins can not rely on the information of who initiated the connection.

So these are the three different options, and none of them are strictly better than the others. You have to accept one of:

• plugins not knowing who connected to them
• plugins not knowing which of the two initiated the connection
• race conditions upon simultaneous connection attempts

I haven't even made up my mind on whether anonymous service consumers are a drawback, or actually the better model (in which case the asymmetric approach would be actually superior to the other ones).

@Aljoscha with regards to plugins, what do you think about doing a depject-y approach where each plugin has a "manifest" that says what services they give and what services they need, then the server connects everyone up. i think this would work best if we assume plugins are specified upfront, including how to start the process, so the server starts each plugin process and then connects each plugin to another as desired.

Ok, I think the symmetric approach is a pretty bad idea. It results in two bidirectional protocols simultaneously running over the same channel, which leads to conflicts. It is possible two design cooperative protocols that can handle that, but you can't rely on that (and it would violate the principle that plugins can talk to each however they like). So scratch that.

What remains is the decision whether a plugin receiving a connection should be given the name (and version) of the initiating plugin. Not sending the name results in a strictly less powerful protocol, so why am I even considering it?

I think relying on the name of the initiator leads to tightly coupled plugins and should be considered an anti-pattern. The whole point of the plugin architecture is to decouple different aspects of application logic. If two plugins rely on each other, then they really are one plugin (note that this is different from one plugin relying on the other but not the other way around). And if plugins really want to treat certain other plugins in a special way, they can still encode this in the protocol they use for communication. But in my opinion, the framework should not encourage this bad practice.

There's a performance impact of not using a single logical channel between the two plugins, but two. Plugins that only use one channel are more efficient, but plugins can only negotiate that if they know their names. But the cost and performance impact of not having an additional channel is negligible, not really justifying the tricky shared-channel negotiation and the necessary compromises for the communication protocol they use over that shared channel.

So I'm leaning towards plugins not knowing who or what initiated a connection to them, even though this is the less general approach. I'd love to hear other people's opinion on this (CC e.g. @keks, @cryptix).

Note that this post only dealt with min-bus, not with a p2p-aware framework. For cross-server communication, there will always be the need to include some information about who initiated a connection (at the very least the public key). But the same core decision needs to be made.

@dinosaur

@Aljoscha with regards to plugins, what do you think about doing a depject-y approach where each plugin has a "manifest" that says what services they give and what services they need, then the server connects everyone up. i think this would work best if we assume plugins are specified upfront, including how to start the process, so the server starts each plugin process and then connects each plugin to another as desired.

Caveat: For now, I'm just exploring the general design space, my opinions are still forming and I've been mostly considering an abstract setting rather than an ssb-specific one.

I'm not a huge fan of a manifest that specifies what kind of rpcs are offered by a plugin, because that ties the whole thing to a specific communication protocol. We don't need to do so, so imo we shouldn't. More interesting is the idea of a dependency specification, where each plugin says which other plugins they want to be connected to (and a minimum version). This is effectively what the models I've been exploring here do dynamically, rather than tying it to a certain file format. I like keeping it dynamic, that keeps open possibilities like having preferences of plugins, e.g. wanting to interact with an ebt plugin, but falling back to legacy replication if no connection to an ebt plugin can be set up.

Specifying what services are offered by a plugin seems rather pointless to me. If you don't know a plugin, you won't be able to interact meaningfully with it anyways. If you do know about it, then you already know what it does. So all such a "manifest" should contain is really a set of dependencies.

The most interesting point you raised is the server starting plugin processes as needed (and by extension shutting them down when they are not needed anymore). That's a very interesting point, and one I definitely haven't made up my mind about yet. In the models I've explored in this thread, you'd somehow need to start all plugins you could possibly need up front, and potentially many of them could be idling without anyone ever connecting to them.

First of all, I don't think idling plugins would be much of a problem, processes waiting for IO are pretty cheap.

Putting this into the spec will greatly increase the complexity. How do you start plugins on another machine, and how do you establish the connection between a newly started plugin and the server? There's a lot of detail hidden there that the server would suddenly need to be aware of. Less problematic, but for completeness' sake: How do you deal with failure to start a plugin?

Another consequence of this: Plugins and processes become arbitrarily tied to each other. In the general setting, a plugin is simply something that connects to a server. There might be a single process that creates multiple connections to the server, and that's totally fine. The server doesn't even need to know about this. There could be some complex logic for determining which plugins to create and how they should connect, plugins could be created dynamically, etc. There are a lot of options that would be prohibited by making plugins server-startable processes.

But my main argument against this: There's no need to do this in the server, this can be done by a plugin. You want declarative plugin management? Go write a plugin that does exactly that, and use it by everything that wants to be part of your declarative world. It can be as opinionated as you like. But everyone else who doesn't buy into your ideas of the perfect declarative plugin setup can happily ignore it, and they don't need to pay the price of an unnecessarily complicated server protocol. This sounds rather dismissive, but I'm not saying that declarative plugin management is a bad idea. I might design such a plugin myself if I think ssb needs it. But I don't want to force my ideas of a good declarative system upon everybody else.

A very related discussion is about some sort of introspection mechanism. D-Bus for example chose to include self-description as part of the spec, thus everything needs to pay the cost (as well as resulting in a much more complicated spec). In exchange, there are generic programs that can e.g. visualize all processes connected to D-Bus, and the services they offer. But you can also get that without bloating the protocol, by creating an introspection plugin. All plugins that want to be introspected could register themselves with that plugin, specifying whatever that plugin allows to specify. There could be a family of plugins that all use muxrpc, and the introspection plugin could allow to share manifest objects, etc. But that still wouldn't preclude other plugins from not using muxrpc at all. If the protocol itself wanted to enforce introspection, it could do so by forcing everyone to use muxrpc and specify manifest objects. But what happens if muxrpc turns out to not be the best format ever? This is what happened with D-Bus, even a tiny D-Bus program will have to supply a bunch of xml manifests. Even if you want to do something conceptionally very simple, you have to do a bunch of unnecessary work to satisfy an arbitrarily opinionated protocol.

TLDR: I'm a fan of kiss (no, not the band).

having preferences of plugins, e.g. wanting to interact with an ebt plugin, but falling back to legacy replication if no connection to an ebt plugin can be set up.

a depject approach could be more than just a dependency specification. rather than saying "i need plugin X", you say "i need interface X" or "i give interface X". we would have a defined set of core interfaces (the core API basically), like connection creator or connection manager or replication or whatever. defining these abstract interfaces will be non-trivial, but if we get our abstractions right then we can have situations where many plugins implement the same interface and we want the first one that works (e.g. legacyReplication and ebtReplication plugins both give replication service, we only need one), or where many plugins implement the same interface and you want all of them as a list (e.g. many connection creators: local network connections, bluetooth connections, pub connections, relay-p2p connections, dht-p2p connections, etc), or where many plugins implement the same interface and you want all of them combined as one (e.g. connection manager could be like this).

rather than saying "i need plugin X", you say "i need interface X" [...].

Isn't this the same? A plugin name is nothing but the promise of speaking a certain protocol. When you talk to a plugin, you talk in that protocol, but you don't care how exactly the plugin does its thing. Caring about it doesn't even make sense, because you can not know what exactly the plugin does - and that's completely intentional.

I get your point, I agree with it, but I think my approach already does this, in a more general way. Just adjust the granularity of what you think of as a plugin. And keep in mind: Multiple plugin names can all be handled by the same process.

TLDR: I'm a fan of kiss (no, not the band).

my approach already does this, in a more general way.

i'm going to ramble a bit, for my own sake and in case anyone finds this worthwhile, feel free to ignore.

so far i've learned simple systems are a balance of cohesion and decoupling. if you are too cohesive (i.e. a monolith), then to extend is expensive. if you are too decoupled (i.e. many micro-modules), then to glue is expensive.

i'm extremely guilty of being the one to implement the most general system possible. what i'm trying to learn is YAGNI, as in, to focus on the problem we're solving rather than potential future problems.

also, the Scuttlebutt project has been guilty of over-modularizing. we made a big mistake with using depject for everything in Patch{core, work, bay}. turns out, you should only use a pluggable module system for plugins, or functionality you want to be extended, not the core functionality.

yes it's true that the most minimal system doesn't need to include declarative plugin management, plugin introspection, etc; yes we could defer those bits to a plugin. but i think if we did bake that into core, where every plugin declared their "manifest" (i mean as a general concept of declarative plugin metadata, not as a muxrpc manifest), where the server started every plugin process using that information, and where then the server provided introspection, that would be less complex overall than the everything-is-a-plugin approach. as in, baking functionality into core (cohesion) means less glue. it's easier if the plugins don't have to worry about how to start themselves (and manage their process in case of failure), how to make sense of connecting to other plugins, etc.

the question is really: will the ability to declare, manage, and introspect plugins be a fundamental feature of our core system?

said another way, is it reasonable to expect that if we don't bake it in to core, we will include a core plugin for that feature.

not to mention, there's an upside to making plugins pay with declarative metadata, yes it is a more restrictive box but also the restrictions allow you to have more clarity of focus inside the box.

it's like that old saying, "bondage sets you free".

I get your point, [...]

Let me rephrase that to "I think I get your point".

Anyways, there are a lot of nice features we could build. But in my opinion, the question we should ask is not "Is this a nice feature?", but rather "What do we gain by making this feature part of the specification instead of implementing it in a plugin?".

@dinosaur

I appreciate the reminder, I guess we'll sketch out both approaches and compare. For now I'll ponder the question of server-to-server connections some more, and how that relates to making things more declarative.

There are a few more arguments for a minimal system though that I feel your post brushes over:

I'm very keen on keeping the ssb protocol as simple as possible. Everything we put into a spec is something everyone after us will have to live with, reimplement and maintain. Also, everything we put into a spec is something that becomes difficult to change without breaking backwards-compatibility.

Moving things into plugins, even if we know that we will always assume existence of those plugins (at least at this point in time), means that ssb itself stays lightweight and flexible. I want to avoid specifying core interfaces, we won't get them right anyways. Putting stuff into plugins also eases the burden on implementors. They don't need to implement everything at once, but they can gradually replace components at whatever pace they want. A monolith has to be replaced in one go.

For application architecture (such as patchbay) it might make sense to err on the side of cohesion, but for protocol design I'd rather want to err on the side of decoupling.

Okay, I managed to get through this entire thread (disclaimer, I just skimmed the part where you where talking about bytes)

My reaction centers around one question: why do you want to have plugins running on different machines? At first I thought this was just a small detail, but then that references to that idea kept cropping up. So as I read on, it started to feel like you are primarily designing a system for building a distributed system across... a whole data center. Swap every occurrence of "plugin" for "mircoservice" and it totally makes sense. I.e. I'm thinking of this as a competitor to zeromq, etc.

This seems like a perfectly reasonable design for a microservice framework, but I don't think that is what we need to implement plugins for secure-scuttlebutt. This is too general and doesn't address very concrete needs that we definitely have (I'll discuss what I think these are in another thread). Also, no one has actually requested the ability to run plugins on different machines. I can't really think of uses of ssb plugins that would warrant that. (Although, I can easily think of non-ssb applications that need that)

On the problem of symmetry: I think this is actually a non-problem. In practice, things are usually clients or servers. Lets say, you have a server that stores file, then clients that get files from it. Clients connect to the server when they need it, the server doesn't go looking for clients. a p2p protocol can have symmetry, basically there are multiple instances of a thing, they connect and replicate, and stuff like that. If you are getting to that scale, I think a bus will become a bottleneck, and you want to make the bus into more of a service registry. But anyway, p2p systems need to be able to deal with lots failures etc, so if A and B both connect simultaniously, you just have a way for the double connection to be detected, and one of them closed, or even, close both, and reattempt a connection after a random timeout.

I think it helps a huge deal, when designing things, to visualize concrete examples that benefit from a specific feature. What is an example of a plausible pair of plugins/microservices that have symmetric connections to each other?

On the subject of a microservice framework, having a uniform way to deploy and run and manage all your services on all your servers is a big part of the offer. That isn't part of the protocol, but it is something you need to start thinking about pretty early on, like, first thing after figuring out the protocol.

Fair enough, the microservice comparison is justified, and I agree that ssb would not really benefit from it. To give a short background on where the idea of plugins running on different machines comes from:

Servers need to connect to other servers, so some part of the protocol will end up to be network-aware in any case. There will be something like multi-addr or multiserver addresses to handle connections. But if we already have that, why arbitrarily restrict plugins to only a subset of those connection modes? That's actually the more complex way to go, not the simpler one. This argument does start to crack once you take a closer look at things. For example it would be very simple to implement an optimization where two local plugins communicate directly via a unix domain socket, but connections between remote plugins could become tricky (NATs etc). So remote plugins are not something I absolutely insist one. I've just been considering the most general setting, so that's why they still appear in later posts.

I fully agree that in a setting where the server has more control (enforcing sandboxing, etc.), it makes sense to distinguish between plugins running on the same machine or not. But in the most general setting, the protocol does not need to care about that.

I can't really think of uses of ssb plugins that would warrant that.

Thin clients come to mind. If clients are actually plugins (and I'm pretty convinced that that is the right way to go, or at least the cleaner solution), then the ability to e.g. run patchfoo on a resource-constrained device while connecting to a more powerful server over wlan would be lost if plugins are forced to be local processes.

Another use-case is something like a really index-heavy plugin (e.g. full-text search over large ssb datasets), or a computationally expensive one. It may make sense to host those on a dedicated server, possibly exposing them to multiple users.

In any case, these could just run a local plugin which then talks to the remote endpoint in whatever way it sees fit, outside the scope of the plugin architecture. That would require asking the sandboxing mechanism for permission, so it involves some complications, but it isn't unsolvable. So these situations are not really sufficient arguments against making plugins local-only.

So overall, I'm open about restricting the scope of this. I'd still like to explore server-to-server connections in the general setting though, before doing so.

@Dominic, I'm curious about your opinion on clients being nothing but plugins that don't expose an API. That's a pretty fundamental decision, especially if plugin-startup ends up being managed centrally.

Making the Plugin Server P2P Aware

For this post I'll retreat into my bubble of the very general setting described in this post. I'll post about sandboxing, plugin startup, and related concerns later.

I see are four basic options regarding a p2p-aware plugin architecture:

• ignore it
• all-or-nothing connections
• scoped connections
• opaque connection management

Ignoring P2P

Plugins are turing complete, so no need to make the system itself deal with p2p stuff. Just handle it in plugins! That's a valid approach, but certainly not what we'll need for ssb. The main draw back is that a plugin would need to reimplement another, nested plugin framework. It needs to mediate connections between plugins connected to different servers. But isn't mediating connections between plugins the job of the framework? Also lots of sandboxing concerns with this approach. So from now on we'll assume it is a good idea to have a p2p-aware framework.

All-Or-Nothing Connections

To recap the setting from the previous post on this topic, each server has its own keypair, and servers can connect to each other, allowing the plugins on one server to call the plugins on the other. Plugins can choose whether they want to expose their functionality to the other server, based on its public key.

That's the abstract idea, but there are a few details needed to realize it. How does a server know when to connect to another server, and which one? When do they disconnect? A server would need to allow its plugins to start end terminate connections. The plugins would need to specify the address of the server to connect to. Addresses would be given in some format like multiaddr, for this discussion I'll just assume that some suitable format exists. Plugins would also need to be aware of the address(es) of their server, so that they can send them across the network.

The main takeaway of this is that the server needs to be aware of both an addressing format, and how to establish connections using that format. Plugins can just pass along (and possibly manipulate) those addresses, but they don't need to care about details of how connection are actually set up.

There are some problems with this approach, including but not limited to:

• all plugins have the capability to set up network connections
• all plugins have the capability to terminate any connection
• what happens if the plugin that was supposed to terminate a connection crashes?
  • Did I mention anywhere that the server should periodically send heartbeat pings to inactive plugins, to detect unresponsiveness and treat it like a crash? Well, now I did. Heartbeats are awesome!
• no support for connections where only a subset of plugins is allowed to talk to each other

Scoped Connections

Here is one (but definitely not the only) approach for dealing with the above problems. When a plugin tells its server to establish a connection to some other server, this does not create a globally scoped connection. Instead, the connection is only accessible to the plugin. To establish such a connection, the plugin must also specify the name of the remote plugin that should handle the connection, and on the other server, the incoming connection is only accessible to the specified plugin. If no such plugin exists (or the version is too low), the connection attempts fails erroneously.

The plugins can then talk to each other, however they see fit. At any point, a plugin can make its side of the connection available to all other plugins connected to the same server. You could do this more granularily with a capability system, but I don't think that's worth it: If other plugins should be able to influence a non-public connection, that can be done via rpcs between the plugins on the same server.

Even after making a server-to-server connection public, the plugin that initiated (or accepted) the connection is still responsible for it. It is the only plugin that can close the connection, and the connection is closed automatically if the plugin disconnecs or times out.

In this (and the previous) model, it makes sense to think a bit about the opacity of network addresses. Since the server implements the connection logic, it is possible to expose connection information (e.g. a mutliaddr) as an opaque object to plugins. But plugins might want to manipulate it, e.g. to not send locally scoped addresses across a network boundary, or to filter out certain connection types. It would be possible for the server to offer these operations. What this gives us is the ability to tinker with address formats without requiring plugins to adapt. But that might simply not be worth it, most likely we'll settle on a particular multiformat and fully commit to it.

Opaque Connection Management

The fourth approach is radically different: Connections are an implementation detail of the server, the server gossips as it sees fit, and plugins can subscripe to connection changes. This would probably be paired with an all-or-nothing model: Once a connection is established, all plugins get to use it equally.

This approach is the one that results in the simplest plugin API, at the cost of moving a lot of complexity into the protocol. It also reduces our flexibility in changing gossip details, and it would mean that the whole plugin architecture becomes tightly coupled to ssb. I currently think that the scoped connections approach is not so complicted that those sacrifices become worth it. The increase in plugin api complexity only affects those plugins that wish to deal with connection setup, all other plugins can simply ignore it and effectively code against the same API as under opaque connection management.

Non-prescriptive connection management will leave open the door for future experimentation. In particular, since the friends graph is not inherently part of ssb, there might someday be other ssb-based applications with vastly different replication needs than our current social networks. As for other arguments for leaving connection management to plugins, the same arguments as those for leaving plugin startup to plugins apply.

Also note that due to connections still being established by the server rather than "out of band", the scoped connections approach plays well with sandboxing, although it will slightly complicate it.

I think this mostly concludes the open-ended exploration phase for me. I'll engage with sandboxing, declarative apis, restricted access between plugins and declarative lifecycle management next, and I'm pretty optimistic we'll be able to find common ground where nobody ends up unhappy.

So let's talk about sandboxing plugins and plugin APIs (mostly ignoring managed lifecycle for now, as that's fairly orthogonal). I'll first sketch an approach that gets around an explicit representation of plugin APIs, and then argue why I think that might make sense.

Sandboxing works through capabilities. By default, a plugin has no capabilities other than allocating memory and maintaining its connection to the server. The protocol defines a set of capabilities, including but not limited to file io, socket io, reading the ssb database, writing to the database, telling the server to establish a connection to another server, talking to remote plugins (possibly filtered by pubkey and/or name), and so on. A plugin specifies which of those capabilities it needs, and if the server grants them (as ultimately instructed by the user), then the plugin is started in a sandbox that only allows those os io actions for which it has capabilities (and the server disallows rpcs for accessing the ssb db etc if those capabilities were not given).

A plugin a may only communicate with another other plugin b if b has no capability that a does not already have as well. And peeking into managed lifecycles: The same condition holds for plugin a starting up plugin b.

That's basically it. The big difference to what I read between the lines of @Dominic and @dinosaur comments/proposals/drafts is that the whole system is agnostic to how plugins actually talk to each other. They don't specify an rpc api, and they don't specify capabilities needed per rpc. A short list of nice things about that:

• fewer stuff the server needs to know about, fewer security-critical stuff the server could get wrong
• does not impose any specifically structured protocols upon plugins
• no need to declaratively specify an API (that needs to be manually implemented anyways, so there's no simplicity gained through declarativity)
• inter-plugin communication capabilities do not have to be specified explicitly, but emerge naturally

In an alternative setting with explicitly named and tracked rpcs, you would get the additional expressive power of assigning different capabilities to different rpcs of the same plugin. That is honestly the only advantage of such a system I can come up with (ignoring introspection, which is orthogonal and which we can discuss at another time). Well, and upon closer examination that advantage disappears, because you can not reliably do this. The plugin as a whole is granted a single set of capabilities, so to fully function, it needs to request the union of the capabilities of all its rpcs. But then, if we don't trust the plugin (and that's the whole reason for the sandboxing in the first place), we can not trust its claim that a certain rpc won't make use of undeclared capabilities. So we end up with a single set of capabilities for the whole plugin, just as in the system I sketched above.

There are a few possible ways of extending the system: Rather than binarily either granting all capabilities or not starting the plugin at all, it could be possible to inform it of the subset of capabilities it was granted. The plugin can then decide whether it wants to shut down or whether it can operate in a restricted mode. Note that this is does not fix the issue about rpc-level capabilities not working: If a plugin is granted all its capabilities, than less privileged plugins still can not be allowed to call methods for which the plugins claims that it does not need some powerful capabilities, because we still can't trust that claim.

As a final note: It could be possible to do dynamic capability (re)assignment, and to define capabilities for talking to specific plugins on the fly. I mention these for completeness' sake, but I think these are bad ideas. The security enforcing parts should stay as simple as possible.

@Dominic I'd love to hear your thoughts on this approach for dealing with inter-plugin communication restrictions, even if it is just a simple "ack".

I'm not fundamentally opposed to enforcing some sort of introspective rpc protocol for plugins (although I would prefer to not do that). There are some good arguments for that (consistency, introspection, stability of the ecosystem), my main point is that capability-based security is not among those reasons.

This post was powered by Felix Mendelssohn Bartholdy. I should listen to music more often =)

Maybe I shouldn't be listening to music when trying to think after all... There are some serious flaw with the previous post. Automatically allowing plugin-to-plugin communication based on capabilities is sensible in a setting where you don't trust any plugins at all, but why would hand out capabilities then. If you hand out a capability to a plugin, you trust it not to misuse it, including not indirectly giving that capability to untrusted plugins through rpcs. So in some sense we could actually trust the claimed capabilities of rpc methods. Placing this trust into plugins allows to give access to some functionality to a less trusted plugin, using the trusted plugin as a filter.

So yeah, much of the previous post is garbage... Sorry about that.

@aljoscha lets discuss possible security aspects of plugin systems in a fresh thread. This one is already a lot to get through, and I think we can consider security aspects somewhat independently of how the connections between plugins are arranged, at least - we can start by considering what can be controlled about a single plugin and work from there.

Tomorrow I'll begin with the #sunrise-choir work, and I'll start out with message formats. So I'll stop exploring the plugin design space for now, but here is a list of open question any solution we'll come up will have to answer:

• are plugins and clients the same?
• can plugins run on different machines?
• should a plugin know the name of another plugin that initiated a connection to it?
• who gets to start connections to other servers? Who gets to close them?
• can plugins run any protocol they like, or do we enforce some sort of muxrpc-like protocol?
• how does automatic plugin lifecycle management work?
• which plugins are allowed to talk to each other?
• what are the cross-platform IO capabilities?
• how do we grant access to the ssb database (capability-wise)?
• which core rpcs are there (gossip, replication, database reads, database writes, blobs, private messages, where does it stop?)?

A few of these are not directly related to plugin management (such as deciding on capabilities and the core ssb server functionality), but they depend on each other enough that we'll likely need to introduce them at the same time. All the message encoding stuff is fairly independent, but rpc protocols, plugins and core ssb functionality are interdependent. Reaching consensus on how to do all of these at once, will likely be the most difficult step in what the sunrise choir set out to do. But also one of the more worthwhile ones, since that is what we need so that clients can code against a spec rather than (j)sbot.

A sandboxing thing we could potentially reuse: CloudABI