having now read this, some notes:

• sbot already provides a manifest method, which is your 'listCommands'.

• I'm with keks on keeping-in-mind need for in-process plugins, at least from the point of view of a specific runtime. separate-process plugins would nearly always work in-process, but not always the other way. Applying long-ago java experience, if the implementation style for a plugin was to implement in-process style, then wrap it with a muxrpc thing, it should be easy.

• indexing: you could have a indexer plugin that other plugins requested indexes from, and you could have a convention that allowed a query module to make use of indexes provided by a query plugin. I think these are both good ideas, not mutually exclusive.

But my most important point:

• instead of a server with unix socket, I recommend sbot manages plugins as child processes, and communicates with them via stdio. This means a trusted core has control over what plugin should be providing what, and I think this will set the ground-work for a really good security architecture later.

@keks okay, lets focus in on the part that I think we agree about, then cycle back around to the question of how the plugins become connected to the system.

subproposal: muxrpc api exposes a "mount" method on the local object (local, meaning: not over rpc), the signature for mount would be mount(name, pluginCall). and pluginCall has the signature pluginCall(type, name, args) (note: name is an array, because methods can be nested) This is the same signature as used internally by muxrpc to make local calls and remote calls. normally, a remote call comes in to muxrpc, and that becomes a call on the local api and the name is used to look up the particular method to call. in the case of a call of a remote mounted method, the call message would just be proxied into the mounted rpc with the first part of the name removed.

Note: in this design, there may be many client/peer connections, but plugins always have a single connection to sbot. plugins cannot see events emitted on sbot, or which client/peer made a particular call. I think this is okay for now. It still gives us quite a bit to work with, and a range of plugins (such as various database indexes) don't need that anyway.

The above proposal is intended to clarify how one might proceed with an implementation, without making a final decision wether to connect plugins via stdio or unix socket. stdio seems really easy to me (having played around with that sort of stuff before) so would be happy to create a demo of that, if the above proposal is agreed.

@keks with this api, it would be easy to implement either pattern.

in pseudojavascript:

return {
  //named this method mountRemote so as not to confuse it with local `mount` method
  mountRemote: function (name) {
    //remote id
    var id = this.id
    //get locally held reference to remote rpc
    var _rpc = sbot.peers[id][0] 
    rpc.mount(name, function (type, name, args) {
      return getPath(_rpc, name).apply(null, args)
    })
  }
}

I disagree that putting the plugin loader be out of "core" makes the small core, because the plugin loader is something that everything interacts with. It is the core. The smallest core possible would be muxrpc server + plugin loader, everything else, including gossip, replication, and database could be loaded as plugins. (although some would need to be in-process maybe)

I dislike the model of plugins managing their own connection, because it is not possible for the trusted core to strongly identify a plugin, and thus it's not possible to have a semi-untrusted plugin. If you wanted to restrict what methods a plugin can access, the plugin could just claim to be a different plugin when it calls the server. But if the core controls actually running the plugin, it can statically identify it, decide what permissions it has (both to RPC and OS).

fs.readFile(path.join(path_to_plugin, 'manifest.json'), 'utf8', function (err, val){
  var manifest = JSON.parse(val)
  var proc = child_process.spawn(path_to_plugin)
  var rpc = MuxRpc(manifest, sbot.manifest())(sbot)
  //XXX oops! not how I proposed it initially!
  sbot.mount(name, function (type, name, args) {
    rpc.remote(type, name, args)
  })
  pull(toPull.source(proc.stdout), rpc.createStream(), toPull.sink(proc.stdin))
  proc.on('close', function () {
    //restart after delay?
  })
}

hmm, okay this made me realize that my initial proposal was insufficient. Probably I'll need to refactor muxrpc so that you can insert things into the localCall method that the stream holds.

@keks yes to another call. Appologies to including implementation specific details, but that is just me thinking out loud. I think the simplest implementation is the ultimate goal, so we need to think about that.

trusted: there is another implementation to this word, it's about how much effort you put into auditing something. By removing the need manage the connection to the local sbot from the plugin, then all that stuff becomes unnecessary, and just finding it there is suspucious. If the plugin connects to the sbot, identifies itself, and then dynamically announces it's api... that is a lot more to audit. The sbot is "trusted code" because we are familiar with it's internals, we wrote it or audited it. A plugin is untrusted code until we read it and decide it's safe... so even without sandboxes we've reduced the amount of untrusted code, in the sense of the code we need to read.

I'm influenced here by thoughts on security after 10 years of qmail (main take away being, reduce security bugs by reducing bugs, and the main way to do that is by reducing lines of code)

@keks managing plugins in a child process seemed pretty easy to me, so I made a proof of concept to demonstrate it. some code here: https://github.com/dominictarr/ssb-plugins2 it's not everything you'd need to run ssb plugin yet, because there is some stuff needed to be able to mount a new api. Anyway, it calls methods on the child process, the child process calls back, the child exits if the parent exits, etc.

oh, that code also depends on this PR to muxrpc

@keks yes, plugins-via-stdio does require us to come to an agreement now, about how those plugins are found and run. But your proposal of plugins-as-local-clients also requires us to come to an agreement now about how plugins connect and set them selves up.

I want to consider all the implications of the proposals: how they might be implemented, what limitations they have, how they may interact, etc. So if you say we don't need to agree about the plugin loader yet, well I still need to think about how that might work. I think the plugin loader is an integral part of the plugin system standard, by deferring that we just invite multiple incompatible plugin loaders.

hmm, I guess with plugins-as-clients you can just run a plugin in a terminal, or have a bash script that starts all the plugins sbot & plugin1 & plugin2 etc. Are these important features to you? It seems to me this might have some advantages during development. But I don't really understand why this appeals to you so much, aside from not needing to specify how a plugin loader works immediately.

yup, sure, see you then.

@luxxor do you have thoughts on this, and would you be interested in joining this call?

hey @Dominic - sorry, i somehow wrote down 8pm our time in my calendar.. but @keks just wrote me that the wtblink was for 7hrs ago ...
how do you feel for the same time +~24hrs later?

Aaactually... +48hrs would work better for me and also fits Keks availablity..

I feel really sorry about this.. was and still am looking forward to this call.

TODO: (private) gatherings to iCal converter

@dominic o/ let us know if you have time. We are available now if that fits.

@cryptix, @keks: @dinosaur, @Piet and me briefly brushed on this topic in our first sunrise choir call. While we can code up pub replacements in Rust, Go, or C, this is an area we will need to tackle in order to decouple clients from js-sbot (and thus allowing truly independent, fully fledged ssb server implementations).

@aljoscha: I'm also coming closer to the plugin topic now. V0 and V1 of my scuttle-shell grant are nearly done.

Do you have notes from your call?

For me the biggest open question is how we want to change muxrpc so that we can indicate the calling feed id, like the X-FORWARDED-FOR header in HTTP.

spec for out-of-process plugins: %a0K84ZS...

just making sure that thread doesn't go under.

@cryptix
No call notes on that. We didn't discuss specifics, only identified this as something that we'll encounter in the future. Piet maybe sooner than me, but I'll certainly chime in once this gets (re)discussed.

As for muxrpc, we want to replace it with bpmux. But the "calling feed id" sounds to me like an application-level concern, not a protocol-level one. Pretty sure we won't need to modify the protocol to implement this.

I designed muxrpc with the use case of an endpoint transparently forwarding messages in mind, so that's going to play pretty well with out-of-process plugins. For example, it allows heartbeats on a per-stream basis, which can be relayed to the responsible plugin, thus allowing plugins to hang without the system getting into an unstable state.

@aljoscha: hmmm maybe I missed the thread, I didn't manage to keep up with all the discussions recently. Does that mean rusbot wont support ssb1 with muxrpc? I'd understand that alot, justh hope we can move everything else forward fast enough.

but the "calling feed id" sounds to me like an application-level concern

yea, I agree but currently in muxrpc there is nothing to hold such info. I could only stuff it in the arguments array which are supposed to be the arguments for the endpoints but that feels way to messy, as well. Which is why I brought it up...

cc @dominic

ps: I guess I designed muxrpc should have been autocorrected to I designed bpmux

@cryptix re: calling id - can you describe why you might need this (describe an application / plugin that might exist, and why it needs such a thing) currently, the identity that owns a connection comes from shs's authentication. you certainly could have some convention on adding fields to the args object (most methods are called with an {} object... but I want to hear a strong case for needing that, because I think the implementation will be more elegant if that feature is left out.

@dominic I think the clearest example would be gossip or ebt, at least the who get's which feed(s) part. buuutt I guess this could also be a helper method inside the muxrpc session...?

Another example I had in mind, which I'm not so sure about anymore would be ssb-ws. Making the websock endpoint termination would be really trivial but handing of the calls to the rest of the system. I'm not sure how to rig that in but now I'm not so sure I'm holding this up right... It's kind of a a plugin in reverse.

but yea, this mostly concernse inter-peer plugins. Most of the app-plugins would be fine without this, I think.

@cryptix @keks good points. yes EBT definitely needs to know who called it, optimizations (request-skipping) and safety features (blocking) depend on that.

I thought that maybe gossip.ping would use that too, but only the client side remembers the ping.
This could benefit from tracking who's called, though.

Hmm, there are several options available to us here:

1. add a fixed field to the call args - so if your method was function (opts) {...} there would now be opts.auth. This would mean that functions taking strings would be unable to know the caller. Also, this mixes application data with system data, which is undesirable, and a security footgun (would need to check that if an application tries to pass opts.auth that it doesn't work)

2. add a field to the call data that is the context: function (opts) { this.id // caller } this is actually how it currently works. For process plugins, the parent process would have to pass this information in with the call, but obviously remote muxrpc get this from the authentication system (shs). A raw muxrpc call message, which creates the request has a json body: {type, name, args} we could just add context to that? should it just be the string? or will we need to add more thing to that ever? (fingers crossed that we don't!)

Hmm, @keks already gave some good examples of why we might need more fields - boolean of whether it was called by self (or a friend etc).