re august ssbcs grants:

Proposal: Adding diamond sockets to scuttle-shell

note: This is a first rough draft to get feedback from the community.

What are you going to do?

I want to contribute a good amount of time to scuttle-shell.
I rolled out the why in the motivation question below but the gist is this: I want multiple clients being able to run of of one sbot instance. Right now we have deadlocks so that you can only run either patchwork or that other client.

An interested circle had a gathering about two weeks ago where we consolidated our vision for what the shell cloud become. The notes from the call notes with a first milestone/roadmap draft can be found on that thread but to make it more specific, here is is my adapted version for this grant:

Milestone #0 - The Basics

1. get these clients working against the shell
   I propose patchbay, tale:net, git-ssb and patchfoo because I use them the most and are familiar with their design.
   working for me means having a functional version of the clients using scuttle-shell installed from npm.
   patchwork will come later as it has specific demands better solved with out-of-process plugins.
   I'd also interested in coordinating with tictack and ngx if they are interested but I don't think this step of the grant should be port all the things but more a show that it works.
2. hard-coded list of plugins
   which is the union of all the required plugins by the above clients.
   currently git-ssb needs ssb-links which patchbay is missing, for instance.
3. add two new plugin load pathways moved to m#1
4. maybe make the API the same as scuttlebot then we could replace scuttlebot with scuttleshell This was my idea and it sounds nice but I fear it will make this more complicated than useful.

notes:
The biggest blocker I see here is getting the platform-dependent systrayhelper binary installed through an npm postinstall script.
With a good multi-platform example/template at hand, I'm fairly certain that I can port/adapt it to our needs.

Then it's just making the clients use the new shell. I'm fairly certain @mix' PR for patchbay is already >50% there and since the code in tale:net for sbot server booting is very similar, that will also be a no-brainer. It's more of a question of polishing glue code once the shell itself is settled.

Milestones #1 - more dynamic

1. be able to get scuttle-shell to take additional plugins
   • maybe be able to add them via providing a path
   • @SoapDog already did most of the required work for this in %8ycfKNk...
2. maybe detect which clients are installed and load their plugins

notes:
I'm not so certain about the useful-ness of 1. but my guess is that it might be very handy for devs to work on new plugins.
I'm also unsure about 2. it sounds like a big chicken-egg thing and yet another level of package management.
On the other hand: we could use this functionality later to multiplex the protocol handlers to specific apps. Like lookup ssb://$someHash and see if it is a type:gathering or type:git-update and delegate to an app that can support this.. now we are totally in dream land again and I'm much more excited about the things that m#2 brings.

Miltestone #2 - use plugins2 to register own plugins at runtime

plugins2 is the working title that popped in my had during the call.
It is about Plugins, which can be registered on an already running sbot.
Right now you need to shutdown sbot server, change the required plugins and start it again.

This makes it possible for any client to supply a plugin that it needs by itself and will allow for removing the too large set of plugins hard-coded in M#0 to near zero once clients migrated to this approach. @keks and I came up with this idea while prototyping our go-ssb sbot and formulated it in %oxtXLx0... after having a call with @Aljoscha. Later we also had a call with @dominic and he wrote up spec for out-of-process plugins.

The gist would be that we can have a really strong core sbot in any language, and clients that want to use map filter reduce in the flavor of ssb-query can just register their needs through this interface. One implementation would then be, starting this plugin as another process and proxying calls through the main sbot when it sees querys starting with that name.
This way our new sbot can support all the existing clients without having to implement any of the query language logic that comes with that specific plugin. Other implementations, like full nodejs, can just require them as usual, with some book-keeping of course.

end part1

My TODO here would be to pick a good example plugin and show again that it can work. My problem is that I'm a bit out of my depth with regards to nodejs coding skills and the code-base as it currently stands. My sub-proposal here would be to form some kind of plugins working group to coordinate my questions and approaches because I fear that I would be digging in the dark otherwise. I had @cel and maybe @luxxor in mind for this if they have interest and energy available. As they are both familiar with js-muxrpcs internals they should be a good fit and I guess cel might also want this for sbotc. I would also add dominic but maybe more as an supervisor/lgtm to not take to much of his time.

At this point it should be possible for patchwork to register it's in tree plugin with ease.

Once we have a working example I'd integrate that feature into go-ssb as well with the aim to support ssb-links so that I can run git-ssb-web off of go-ssb, if we get to that point.

How are you going to spread your 1 month equivalent of work?

I'm having a hard time specifying a fixed time-frame with starts and ends and would much rather say I will use this as a time-bank and drain it till it's empty. I also worked on my go-ssb grant like this, even though it probably went empty before we reached our self-proclaimed finish line.

What I can say is this: I won't be able to start to work on this before the 3rd week of August and I won't be able to work full time on this but aim for part time / around 20hrs per week because I have other obligations to attend as well. If you don't know me well (yet), read my new-people post to see where I'm coming from.

Why are you motivated to do this?

As one of the developers of the tale:net client (See %q6Im24T..., #talenet and https://t4l3.net/concept/ if you want to learn more about it) I am a tiiiiny bit frustrated that we currently have to tell our users you can only use patchwork OR tale:net at the same time. although the database design would allow for them to be running at the same time. It also pushes us into a direction where we think about adding features much better suited and already done by other clients.

In short: I’d like to work on a project that merges the scuttleverse closer together instead of making new islands.

I think @mix also shares this frustration, being a developer of multiple clients. When he made a PR for patchbay to integrate scuttle-shell, I immediately tried it on a strict linux and found some problems with one of the dependencies of the shell, which I found solutions for. That thread might also be an good example of how I share my progress if you haven't followed my #go-ssb work.

@cel also helped and tried to see how well it works with patchfoo, which it did, after some fixing around the systrayhelper.

I also talked loosely to @SoapDog about this proposal and he welcomes it, especially because then he isn't alone with it.

In general I'm very motivated to work on this because it felt like a group effort from the beginning.

Closing note

I hope didn't forget links or context and hope to convey all this technical stuff at high enough level for everybody.
At least the motivation should be understandable for less technical people.

Looking forward to your feedback and suggestions!!

Again if that wasn't clear from the how part: I'm not super-pressed to get one of the August slots but wanted to get the discussion around this started. I am in need of a new job around September, though and this kind of money would float me for around 5 month.

ps: the diamond sockets are a metaphor for slots where the plugins sit on the shell, making each hermie unique and shiny :)

Am I imagining a coordinated team effort here? Or is it really happening?

@bobhaugen: Honestly I'd say not yet but it felt like we gathered around a problem that we all have, instantly to me at least. Additionally I think this needs eyes and mindshare of a lof of people to become actually adopted. And if it goes roughly forward as I envisoned it above, we have all necesarry parties involved, low-level and client devs. so producers and consumers pulling on one rope.

thanks @andre(sphone)! It's always encouraging to hear that it's not just ones one weird thought..

might not be the expert to consult in it, but feel free to toss some questions at me and I can function as a rubberduck for debugging

sure will : ) I understand it quite well, to a point at least since I implemented the wire format in Go. What I'm missing understaning of is the actual handler / dispatcher implementation in JS that does the call invocations, which needs to be changed to support the proposed proxying. Abstractly I already have it in my head, just forwarding by prefix match to the sub-process which is another client-server muxrpc session.

On that note: there is another important TODO wrt plugins2: we need to somehow pass through the actual caller keypair when doing proxying. Currently the shs pubkey is passed along and we need to forward it as well when proxying calls from a remote peer to a plugin. Otherwise the plugin might think that it is doing something for the local ID instead of the remote one.

And one more note regarding your note: it would be good to keep in mind how could scuttleshell manage multiple identities. I know this is a stretch, so don't consider it an important requirement, I just want to bring to attention that multiple identities (personas) for a single user opens up new application use cases, so keeping this in mind might influence the architecture to not hardcode a singleton identity.

multiple identities definetly is an interesting proposition! If it derails too much I'd like to fork it off this thread, though.

Just to be clear: do you mean multiple personas at the same time?

If so: I vagely recall @dominic saying that core sbot already supports this but just needs some more batteries. I'm not sure if this was about publishing or querying. Also I'm unsure if this can be fed into clients from the shell or how much clients need to change their assumptions for this. I think most will assume my user is what ever is in .ssb/secret.

If not, It might be interesting to consider a simple switcher, though. Having a menu in the systray to switch form one persona to the next and having sbot reboot.

This sounds like a very good proposal. My only question for now is if this is tied to having something running in a tray or not?

I really like this proposal for multiple reasons, the most important are:

• The sustainability of the scuttle-shell approach. The more people improving the source code and getting familiar with it the better.
• The benefits for the overall ecosystem as apps can focus on implementing novel features without the need of ssb boilerplate.
• Many apps alive in a device at the same time can lead to some quite innovative apps.

I will happily play the cheerleading role of waving my pompoms and cheering more developers forward into this project!!!! :-)

@cryptix yes, we have multiple identities, I had started on it, then @the-dark-lord finished it up, and made some PRs, https://github.com/ssbc/secure-scuttlebutt/pull/213 but again, the front ends now need to catch up

I'll loop back to details
but wanted to drop in to say : this is ultra strategic - this is a step in the direction of very rich ecosystem... a world where you can have all the clients open and use them each for their strengths, a future where people can write plugins in different languages

+1 +1 +1 +1 +1

Totally back the idea and the human

My only question for now is if this is tied to having something running in a tray or not?

thanks for the question @arj! I suppose you don't like trayicons? May I ask what aspect of them you find anoying? I personally find try icons more bareable than terminal windows with long running processes...

unfinished thoughts: node/npm doesn't have something like conditional installs or build flags, right? The idea would be to have tui uropod and tray uropod modes to get the same thing, functionality wise.

With the current use of the Tray API it might be doable to shim an alternate UI like below but at some point it might be weird to support both...

$ scuttleshell

Welcome... 

Actions:
(1) Do this
(2) Do that
(3) Toggle foo

> please choose your action:

Personally I think scuttle-shell should add next to nothing to what stock sbot can do on it's own. Meaning: people prefering to run sbot server manually should still be fine but they might need to tinker with installing plugins on their own. The oop plugins api will go into sbot not scuttle-shell. So everybody should be fine in the long run.

Perfect, thanks for clearing that up. I really like this proposal btw, just wanted to clarify where exactly one starts and the other ends.

@cryptix this looks good, but I don't understand what "diamond sockets" refers to?

@dominc hehe it’s more of a cutesy joke. See the last line of the 2nd post. After Nanos comment I might call it uroplug.

Just to link back some progress.

M#0: The biggest blocker I see here is getting the platform-dependent systrayhelper binary installed through an npm postinstall script.
With a good multi-platform example/template at hand, I'm fairly certain that I can port/adapt it to our needs.

This is now done. We have a new module forked-systray (gh, git-ssb) with a cleaner install story then the original version. I also patched scuttle-shell to support tale:net, patchbay and git-ssb against an sbot spawned by patchtron, demoed here.

For the last weeks I tried to work 2,5days on the project per week (two 8hr days and the rest sprinkeled between the lines) but this turns out to be quite frustrating for me. It feels like I just got my head in the right space to continue making progress before I have to drift of to my other job. Thankfully the other project is also flexible and will mostlikly understand my wish to change this, as the same problem effects it as well.

Therefore I will change my shedule to work one project per week and alternate them weekly. I hope these prolonged slots of fucus will help also with my availability and responsivness for others.

Therefore: A totally unbiased coinflip decided to let #scuttle-shell sleep next week. I'm looking forward to prototyping an out-of-process plugin the week after.

cc #dev-diaries @SoapDog @mix

@cryptix Piet and I have found the same challenge in doing grant + other project, I think your observation and response are smart. Look forward to hearing how it goes for you

started drafting chapter 2 of plugins

Finally made progress on works in installer/AppImage: %P9fyFLu...