I appreciate the reminder, I guess we'll sketch out both approaches and compare. For now I'll ponder the question of server-to-server connections some more, and how that relates to making things more declarative.
There are a few more arguments for a minimal system though that I feel your post brushes over:
I'm very keen on keeping the ssb protocol as simple as possible. Everything we put into a spec is something everyone after us will have to live with, reimplement and maintain. Also, everything we put into a spec is something that becomes difficult to change without breaking backwards-compatibility.
Moving things into plugins, even if we know that we will always assume existence of those plugins (at least at this point in time), means that ssb itself stays lightweight and flexible. I want to avoid specifying core interfaces, we won't get them right anyways. Putting stuff into plugins also eases the burden on implementors. They don't need to implement everything at once, but they can gradually replace components at whatever pace they want. A monolith has to be replaced in one go.
For application architecture (such as patchbay) it might make sense to err on the side of cohesion, but for protocol design I'd rather want to err on the side of decoupling.