I picked up plugins2 from %0DLS7O6... and added simple go plugin using the same stdio consturcion.
worked with no thrills :]

Now looking into where I could plug this into muxrpc/sbot to achive the redirection to those plugins. I think this could be muxrpc agnostic for the most part but I'm not sure who to achive the catch all/registration yet.

@dominic can you give me some pointers how and where you think the registration should happen? I tried to read the muxrpc code but I just can't see how it works... more precisly: how should can I register an endpoint that gets all the calls..? Otherwise I don't see how to dynamically register new plugins at runtime.

Otherwise, I guess this is what you meant previously by load them first, like collect the wanted plugins with their manifests and then register that concated/merged manifest to muxrpc?

Answering, my own question: I prototyped a small loader and I think the later is the only supported way right now.

quite happy with the mess I came up with!

I made a fork (git-ssb) of ssb-plugins2 which now exports a load(pluginPath) method that can be used to create a ssb-pluginified out-of-process plugin and hacked it into the @cryptix/plugins2 branch on scuttlebot (gh/ssbc and git-ssb).

I chose to reuse the existing plugins code to get the name and collision
handling. Now, if you specifify a plugin as an object instead of a bool in your config,
it will use my ssb-plugins2/load to spawn it.

{
  "plugins": {
    "normal": true,
    "ooplug": { "location": "/abs/path/to/plug" }
  }
}

working:

• plugins spawn
• its manifest is part of sbots

bugs/todos:

• calls dont execute. i guess the name prefix has to be stripped for calls to register correctly
• load: passing sbot into the muxrpc/api results in a circular definition and stack explosion

@cryptix that looks great!
question: is the path a path to an executable? I was considering maybe it should be a path to a directory, with a executable and manifest inferred by a convention, OR, have a path to an executable + manifest. Hmm, have a think about how to specify it's permissions?

When I was writing this, I remember thinking that a plugin wouldn't "know it's own name" it would export {foo, bar, baz} and then get loaded as the fbb plugin. calls to sbot.fbb.foo would get routed to fbb, but fbb would be stripped from the call path before written to the plugin. so the sbot receives fbb.foo but the plugin just sees foo. I think this is important for security and for modularity, because it gives the server the ability to rewire plugin dependencies.

I think it would be good to have a plugins2 method to load/reload/kill a plugin on demand, though I think I recall only implementing plugins at startup?

@dominic glad you like it!

re paths: it's the directory in the config. the loader than looks for a bin and manifest.json inside the folder. I chose this but am not ultimatly happy about it since it would also be nice to expose the version of the plugin to the loading sbot. so maybe moving the bin into the main of package.json would make sense overall?

re prefixes: yes, I also think it should happen that way. The manifest/api is already returned like that but I'm not sure where the prefix-stripping should happen. I guess the run.js needs to use a wrapped packet-stream-codec? Though touching that with a mapper on each packet sounds very perf intensive...

another thing I don't really understand from the code I copied from you is this (also in run.js now):

// in practice, the localCall method is created
// from the local api and manifest
function localCall (type, name, args) {
  console.log('CALLED', type, name, args)
  var cb = args.pop()
  cb(null, { okay: true })
}),

I don't know how it fits into the picture, or what its job is in the muxrpc stream construction.

I also made the package npm testable but its not asserting anything yet, just seeing that it can run the example plugin.

I think it would be good to have a plugins2 method to load/reload/kill a plugin on demand, though I think I recall only implementing plugins at startup?

yes, I agree, realoading crashed ones would be nice! From what I've seen, I'm not sure the manifest can be updaed after the inital load?

@cryptix you don't have to remap every packet, just the packets that start a new request/stream. After that, you would just have a table to map the request number on connection to another. If you wanted to get really tricky, you could just take the incoming buffer and rewrite the request number, then send it. But I expect for security enforcement, it may be desireable to do some packet inspection. Although, probably it should mainly be done on the first packets though, since that is where the arguments are set. Then you could for example, have a plugin that is allowed to publish new messages, but only a given type.

localCall is a method that get called with the raw data from a new request, and it's responsible for gluing muxrpc to your actual api. so type is stream/async/sync, name is the path to the method, and args is the arguments to call with. It either calls the function & callback or returns a stream, or throws an error.

Okay, good but than I‘m conceptually on the right track. I know about the packet structure but it sounded wrong to touch every packet. So I will try to create that request mapper today..! :3

geez.. okay! Calls work now!

I just forgot to split the name into an array (passed ['plug.name'] instead of ['plug', 'name'])

now constructing a more complex example and thinking about better testing.

on the plus side: I now know where to put debug prints in packet-stream-codec to see all the stuff.

@dominic: I picked this back up today after seeing your reply in matts post but I need your help with plugins calling back to the plugin-host and it's API.

It would be really great if you could look into this PR and help me get test/load.js to pass. If I try to pass in the server in load.js to be able to call the hosts apis (like whoami or getAdress from a plugin) everything goes bonkers.. the returned manifest is definetly wrong but I just don't understand what to do or where that packet switchboard would go that you described...

After we solved this I think I can revive this branch on sbot! I wanted to add tests for this way to load plugins but currently have problems getting the sbot test suite to pass... There still seems to be a problem around mutliserv@5 and it picking the wrong default for localhost. see my comments here: https://github.com/ssbc/scuttlebot/pull/566#issuecomment-447834469 - I also have this behavior on my debian patchfoo installation when I updated from 13.0 to latest. It only listens on the public v6 while sbot whoami tries to connect to localhost:8008...

@cryptix sorry I let this slip. I got load.js to pass, didn't have to touch any networking stuff though.

Okay.. Interesting.. Thanks a lot for looking into it!!!

Still can't get all the sbot servers to pass but I made a PR and added a oop-plugin test.. but I still get the stack explosion: https://github.com/ssbc/ssb-server/pull/605