More vague addendums:
- If the server is to mediate direct connections between plugins, it needs to know what kind of direct connection a plugin supports. Since a plugin might live on another machine, the server needs to know, whether that server has e.g. a public ip address, or whether hole punching is necessary. This can be solved in a very simple fashion: Wen a plugin registers at the server, it specifies a set of multiserver addresses that can be used for setting up direct connections. This way, it would even be possible in a setting where two plugins share a machine but the server lives on a different machine for the server to mediate a unix domain socket connection between the two plugins.
- When two servers connect, they'd then query for each other's plugins, and then establishing connections between those plugins based on the result of the query. As an optimization, it should be possible to send messages saying "If you have the foo-plugin, please use one of the following multiserver addresses to establish a connection to my bar-plugin.". That saves a roundtrip before the plugins can talk to each other.
- Just as a clarification: In a setting like that described above, if plugins live on the same machine as their server, instead of setting up a new physical connection, the connection between remote plugins would be multiplexed via bpmux over the original connection between the servers.
- With this plugin architecture, pretty much all functionality of an ssb server except for plugin setup itself could be provided by plugins. This would reduce the pressure on us to figure out a set of "core rpcs" everyone has to support, instead there'd be a few orthogonal core plugins. Since servers exchange information about which plugins are present, this would serve as a kind of feature announcement. This avoids roundtrips just for finding out that the other endpoint does not support ebt, because you'd only be able to send rpcs to the ebt plugin if it was available. If a server finds that the peer does not support any plugins for them to replicate with each other, they can simply drop the connection. As a consequence of this, we would not need to specify mandatory core plugins, this could instead evolve dynamically.