Making the Plugin Server P2P Aware

For this post I'll retreat into my bubble of the very general setting described in this post. I'll post about sandboxing, plugin startup, and related concerns later.

I see are four basic options regarding a p2p-aware plugin architecture:

• ignore it
• all-or-nothing connections
• scoped connections
• opaque connection management

Ignoring P2P

Plugins are turing complete, so no need to make the system itself deal with p2p stuff. Just handle it in plugins! That's a valid approach, but certainly not what we'll need for ssb. The main draw back is that a plugin would need to reimplement another, nested plugin framework. It needs to mediate connections between plugins connected to different servers. But isn't mediating connections between plugins the job of the framework? Also lots of sandboxing concerns with this approach. So from now on we'll assume it is a good idea to have a p2p-aware framework.

All-Or-Nothing Connections

To recap the setting from the previous post on this topic, each server has its own keypair, and servers can connect to each other, allowing the plugins on one server to call the plugins on the other. Plugins can choose whether they want to expose their functionality to the other server, based on its public key.

That's the abstract idea, but there are a few details needed to realize it. How does a server know when to connect to another server, and which one? When do they disconnect? A server would need to allow its plugins to start end terminate connections. The plugins would need to specify the address of the server to connect to. Addresses would be given in some format like multiaddr, for this discussion I'll just assume that some suitable format exists. Plugins would also need to be aware of the address(es) of their server, so that they can send them across the network.

The main takeaway of this is that the server needs to be aware of both an addressing format, and how to establish connections using that format. Plugins can just pass along (and possibly manipulate) those addresses, but they don't need to care about details of how connection are actually set up.

There are some problems with this approach, including but not limited to:

• all plugins have the capability to set up network connections
• all plugins have the capability to terminate any connection
• what happens if the plugin that was supposed to terminate a connection crashes?
  • Did I mention anywhere that the server should periodically send heartbeat pings to inactive plugins, to detect unresponsiveness and treat it like a crash? Well, now I did. Heartbeats are awesome!
• no support for connections where only a subset of plugins is allowed to talk to each other

Scoped Connections

Here is one (but definitely not the only) approach for dealing with the above problems. When a plugin tells its server to establish a connection to some other server, this does not create a globally scoped connection. Instead, the connection is only accessible to the plugin. To establish such a connection, the plugin must also specify the name of the remote plugin that should handle the connection, and on the other server, the incoming connection is only accessible to the specified plugin. If no such plugin exists (or the version is too low), the connection attempts fails erroneously.

The plugins can then talk to each other, however they see fit. At any point, a plugin can make its side of the connection available to all other plugins connected to the same server. You could do this more granularily with a capability system, but I don't think that's worth it: If other plugins should be able to influence a non-public connection, that can be done via rpcs between the plugins on the same server.

Even after making a server-to-server connection public, the plugin that initiated (or accepted) the connection is still responsible for it. It is the only plugin that can close the connection, and the connection is closed automatically if the plugin disconnecs or times out.

In this (and the previous) model, it makes sense to think a bit about the opacity of network addresses. Since the server implements the connection logic, it is possible to expose connection information (e.g. a mutliaddr) as an opaque object to plugins. But plugins might want to manipulate it, e.g. to not send locally scoped addresses across a network boundary, or to filter out certain connection types. It would be possible for the server to offer these operations. What this gives us is the ability to tinker with address formats without requiring plugins to adapt. But that might simply not be worth it, most likely we'll settle on a particular multiformat and fully commit to it.

Opaque Connection Management

The fourth approach is radically different: Connections are an implementation detail of the server, the server gossips as it sees fit, and plugins can subscripe to connection changes. This would probably be paired with an all-or-nothing model: Once a connection is established, all plugins get to use it equally.

This approach is the one that results in the simplest plugin API, at the cost of moving a lot of complexity into the protocol. It also reduces our flexibility in changing gossip details, and it would mean that the whole plugin architecture becomes tightly coupled to ssb. I currently think that the scoped connections approach is not so complicted that those sacrifices become worth it. The increase in plugin api complexity only affects those plugins that wish to deal with connection setup, all other plugins can simply ignore it and effectively code against the same API as under opaque connection management.

Non-prescriptive connection management will leave open the door for future experimentation. In particular, since the friends graph is not inherently part of ssb, there might someday be other ssb-based applications with vastly different replication needs than our current social networks. As for other arguments for leaving connection management to plugins, the same arguments as those for leaving plugin startup to plugins apply.

Also note that due to connections still being established by the server rather than "out of band", the scoped connections approach plays well with sandboxing, although it will slightly complicate it.

I think this mostly concludes the open-ended exploration phase for me. I'll engage with sandboxing, declarative apis, restricted access between plugins and declarative lifecycle management next, and I'm pretty optimistic we'll be able to find common ground where nobody ends up unhappy.