%1bz0TXDuaM65KMTb8bgtrXuqD7L77baneTdoJ0EwRug=.sha256Why it is so hard to build SSB apps 
The dream
As @mix describes in Proliferate the Apps, the great advantage to having many different apps for different things is that we don't need to agree on what goes in apps, and that people are free to experiment with all sorts of different uses of SSB.
We can imagine SSB as a protocol like http, where many different kinds of apps can be built on top and interact together.
The Coordination Problem
But for this to be a reality, all of these apps must agree on a set of constraints and implement a compatible interface that allows all of them to run simultaneously and access the important data without clobbering each other.
Right now, apps can't run simultaneously because they can't agree on which plugins to run.
The story of running apps one-at-a-time is only slightly better. Sure, this way they can choose how they interact with sbot and what indexes and plugins they use. But with the current system, apps are not required to leave the place as they found it. They don't respect the commons that is the .ssb folder, clobbering indexes, and potentially even corrupting the database (e.g. the gossip.json file went awol recently).
One way to solve this is to be much clearer about what apps are allowed to do in the .ssb folder, or run some kind of super process (e.g. scuttle-shell) that is responsible for all of the .ssb interaction. A gate-keeper. But this is really what sbot was already intended to be in the first place. We are always going to run up against compatibility problems as long as we are trying to share the same backend.
So why are we trying to share the same backend?
Shared identity
I think the root of this constraint is that we want to be the same "person" in every app.
Much like the inspiration behind OAuth, many people do not like the idea of having a different "account" with every service they use. It is quite nice to be able to tie your presence on twitter with that on GitHub, or Gitlab for that matter!
To use the same identity in every app, we need something to synchronise the feed so that the hash-chain does not fork. We require atomic writes.
Shared data
There's also the fact that there is a lot of crossover between apps.
Patchwork and Patchbay are both windows to the same people and data. They share a similar purpose, they just allow you to interact with that data slightly differently.
This is a wonderful thing that we have not seen much of on the web lately. The closest thing the web has is APIs, but in those cases, the data is owned by one app and is "allowed" to be used by another. This access can be revoked at any time (think twitter, or soundcloud APIs).
Shared storage
A possible way to avoid some coordination requirements in sbot (less things to agree about), is to use separate indexes for every app. This way, each app would have its own set of views on top of the main log. But the problem is that potentially takes up a lot of extra space.
So far, the general consensus has been that we should agree on what things should be indexed, and then share these with all the apps so that we don't fill the user's hard-drive up unnecessarily.
How can we make this better?
This is a very difficult problem to solve while preserving all of the above constraints. I think that the current requirement for everyone to agree on what goes in .ssb and how this is accessed is dramatically hindering development of new ideas on this protocol. I think that the level of coordination required is the number one problem with SSB right now!
The great thing about ssb is that all of the peers that make up the network do no need to agree about many things. There is no singleton state. Everyone sees the network slightly differently. However, this does not apply to apps running on the same machine.
In order to move past this current bottleneck, I think we're gonna have to make some concessions to the requirements (shared identity, shared storage, shared data) listed above. We need to decouple the clients from each other.
Decoupling the scuttle stack
There are varying degrees of this.
Every app is a peer (nobot)
The most hardcore version would be to give every app its own complete scuttlebot, with a separate identity and log. But the problem is that we give up all of those awesome advantages of shared identity, data and storage. Individual app development would become wonderful, but we'd have all the problems of the web and APIs (patchwork would own certain data, and other apps would have to request it if they wanted it).
We could probably solve the shared identity problem with sameAs. However, solving the shared data problem would expose us to lots of storage duplication, and/or lots of inter-app API stuff (like the web of old).
A very minimal shared scuttlebot (lessbot)
In this case, you would still have a single sbot, however all that it would contain would be replication and the main log.
It wouldn't have queries or views. These would be done in the individual apps. Apps would probably read the log directly for indexing, and for writes, via some kind pipe directly into the process.
With this you get the shared identity and some amount of shared data/storage. There would still be some duplication of data, since each app would have its own indexes. However, the best part about this option is that the only thing that apps need to coordinate about is "writes". Reads would all be done straight off the clients local index (and taking advantage of the append only nature of the shared log, could just use offsets to read directly).
Allow clients to register their own plugins at runtime
This option is the closest to how things work right now, but it still requires all apps to agree on a lot of things (for example, sbot version). This would be good option if we could get it to work, bit that in itself will be a massive coordination exercise. What about multiple plugins interacting with each other? Different versions? On the surface, it sounds trivial, but start digging in and it's really just all the same problems as we have now, just a little bit deeper.
And that's why it's hard!
The current system makes it hard, but creating a better system that makes it easier, is much harder than just putting up with the current system.
I am personally feeling quite motivated to move away from the shared sbot (this coordination stuff takes a lot of energy), but I am also all too aware of the potential ecosystem that could grow out of SSB if we can get apps to work well together. We need to strike the right balance!
#patchwork-dev #apps #scuttlebot #scuttleshell #flume #against-consensus