...continued from previous post

Connecting plugins

Plugins need to be able to interact, for that we need to reimplement dbus provide a few basic facilities. Each plugin should be able to:

• query which plugins are currently connected (by well-known name)
• be notified whenever a plugin connects/disconnects (by well-known name)
• create a bidirectional communication channel to another plugin (you guessed it, by well-known name)

The last one is the big difference between name-spaced plugins and global plugins, and I think it is the cleaner solution. Not only does this allow plugin-to-plugin communication to use any sort of rpc protocol they want (in particular, a fully-fledged rpc protocol might be overkill), but it also allows direct connections. The simplest implementation for the server is to hand out a logical channel to the plugins, which first sends data to the server and the server then reroutes it to the other plugin. But a more advanced implementation could instead mediate creation of a direct channel between the two plugins. On the same machine, that could mean creating a unix domain socket and would save a few memcpys and syscalls. But for plugins living on different machines then their server, this would mean a direct tcp connection rather than (literally) rerouting everything trough the server. This is another advantage we just could not get with anonymous plugins.

There are a some details to work out for this, and there are some considerations regarding the capabilities required for querying connected plugins and establishing connections, but overall, this is surprisingly simple.

Plugin startup

If the server has to start plugins, it needs to know a bunch of things about them, and it also needs to decide when to start them. What is a design where the server doesn't need to care? It can expose well-known sockets (unix domain, tcp, whatever transport you want to use) where processes can register themselves as plugins. "Registration" sounds fancier than this needs to be, at the bare minimum the plugin would just say "Hi, I exist, and I go by the name plugin_foo." and the server would respond "Cool, here's a bidirectional communication channel we can use for all remaining stuff", while notifying subscribers that the plugin connected.

We'd probably want to perform a secret-handshake first, so that the connecting process (which may or may not live on the same machine) proves that it has the capabilities of a certain identity (i.e. the secret key corresponding to a feed id), which may or may not be the same as the server's identity itself. No need to encrypt the connection if it is on the same machine, but it still makes sense to identify via shs.

Does this sound familiar? It is how clients currently work, except for the part where a plugin specifies a name.

With that done, there's a dedicated, bidirectional communication channel between the server and the plugin. At the very least, the plugin should be able to call all the rpcs that a client can call. But in addition to that, there's the plugin-to-plugin communication as described above, facilitated by the server.

Unifying clients and plugins

Wait, what? Well, with this approach, plugins are basically a generalization of clients. A client is simply a plugin which not offer rpcs to other plugins. That's why it does not need a name. So all we need to do is to allow nameless plugins which do not show up in the plugin query/notification mechanism. To the server, there's no distinction between clients and plugins anymore, which will simplify a lot of things.

Remaining thoughts

(Jotted down in a hurry)

I brushed over authentication details: When opening a channel between two plugins, they should each know the public key of the peer (and the server needs to enforce that they have the corresponding private key).

Also, it would be possible to take a much more opiniated approach, forcing all plugin comunication to happen via bpmux, etc. This might help with consistency, but overall plugins still need to come to agreements, so they might as well agree on the encoding of the communication mechanism. In practice, prettty much everything will likely default to bpmux anyways, and those who don't - well, they'll have good reasons and be happy that we didn't enforce it.