You are reading content from Scuttlebutt
@Dominic %r+o8bQTztZrWkY4l2e3iHzgH3MnW1NddYDQpSecrpLE=.sha256

ssb reccomendations

I'm not working on ssb anymore, so those of you that are can take this with a grain of salt. however, I have been composing this post in my head a while and the day that musk's emmanent takeover of twitter is announced seems fitting.

tl:dr; take the best parts of ssb and make something more minimal

new feed format

Create a new client with a new feed format, that's efficient and binary and not json. This would be a breaking change, or rather - current clients can't replicate it. You could do this and implement backwards compat later, but just start with something new.

remove json from protocol

json can stay in sbot-client apis, but remove it from the p2p protocol. Replace that with binary protocols that are easy to implement across languages and efficient.
ebt is one of the technically best parts of ssb, refactoring it to do binary wouldn't be hard.

new feed format uses lipma

Maybe it doesn't matter so much if you don't have all the messages ever. It's certainly a stumbling block for ssb. It's was an elegant design in principle but isn't really what people want in practice. lipma-links mean you can still get everything but don't have to. that means you can have a initial sync that is always fast, because it just replicates what it can within a given time.


I think ssb focus on a single social network has held back developer adoption. I wanted ssb to be a comprehensive id system, where everything else fitted into one system, but it was/is too early for that. It would be better to make a base protocol that was easy to build into other things, independently and not try to make a unified identity system. This was an approach that hyper/dat got better.

with ebt, ssb is the most efficient protocol for replicating a dataset with writers in the thousands. it can also support non-overlapping datasets. but not many things benefit from these strengths.

To make ssb a comprensive decentralized platform were everything is integrated needs means that an ssb app needs to be some sort of browser thing and that's a whole other omega project. But being easy to embed ssb into an app doesn't prevent that.

@Rabble %Wl2SKK9pZXONoNZR18ldEswSj9XPQtUqEA4UMNvqZ1c=.sha256

I'm curious what you think of #p2panda as it does a bunch of the things you're talking about.

@Dominic %gz7SQuqSN2yCubwgdKYp+qnnNDkeoLSFdwXxcclQmJ4=.sha256

sounds like p2panda needs to use ebt (with request skipping) it's means that when you have thousands of logs, and they don't havn't changed (which is the usual case) then you don't send a large handshake. It makes connecting and reconnecting cheap, rather than expensive. A decentralized network with EBT/request-skipping is same big O for data as it would be with a centralized service (sends each message once + fixed overhead)

User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
@Dominic %pWnnWx1Fe4mGWcvzFLypEmT3pC6avcioR5T0afNwwyQ=.sha256

@TheTaoOfSu (mobile) I did actually just suggest a hard fork to get rid of json. sure that goes for other javascriptisms.

@Dominic %sXFs7l5Sl9TlJKkEc2UM8+CzINecci8pL8603xdDaBc=.sha256

remove js from the protocol: not just messages, but calls to replicate via ebt, calling muxrpc passes arguments as json.

as to what format to encode the messages in... if you were gonna refocus it as embedded into applications (rather than as the host to applications) then the message format should be chosen by the application, and it should be able to be schemaful, which can significantly improve both performance and compactness.

what is inside the messages doesn't actually matter to the replication protocol, so we can put that aside for now

@Dominic %Cv1cukBDP+T4mbSa7LaLzUv/nzsqg9fYL8IZW5SpOl4=.sha256

The parsing format is not the hard part. Even if you had to hand write the json parser it wouldn't make that much difference to the overall implementation effort.

I once wrote a JSON parser in bash it took maybe a day (and because it went viral and got some good PRs it's actually a pretty good JSON implementation... also very slow, because it's bash, wow that was 11 years ago)

It's not just json per se, it's how you work with it. parsing it into another data structure that then gets GC'd it adds a lot of bloat when you can have a parseless format (for example bipf, but if you use a schemaful format it's much better again)

@andrestaltz %FndwnRvWTwW16Kur6pQ0fPdTtMc8uCuCl11jbwiLCMs=.sha256

As other people have said, it seems like you're suggesting p2panda + EBT.

I'm not so keen on breaking changes, I think it's completely non-trivial (almost mystical) how to get traction, and especially high quality traction. SSB at the moment has that. For some reason, we have a great and authentic community that keeps growing organically, and this has to take priority over any other technical improvement.

That said there are lots of ways of going forward which will mean breaking changes, but at least we can deploy them gradually and piece-wise, instead of the dramatic big rewrite.

new feed format

I'm not sure if you're familiar, @Dominic, with our work on partial replication, but one of the things we came up with was metafeeds and they are encoded in a new binary feed format we called bendy-butt, because it uses bencode (from bittorrent).

One of the great advantages of the metafeeds design is that it will allow creating subfeeds that use any feed format, so we're already preparing this system so that you can have a gabby grove subfeed or a bamboo subfeed. To some extent this support is already verified in unit tests.

ebt is one of the technically best parts of ssb, refactoring it to do binary wouldn't be hard.

If I'm not mistaken, @arj and @cryptix worked on updating EBT to support binary. It's here: https://github.com/ssbc/ssb-ebt/tree/master/formats

new feed format uses lipma

Yeah it would be possible to have a subfeed of a metafeed that uses bamboo, or something else with lipma links. Similar to how multiserver supports upgradability, I think it's important that we allow several feed formats, instead of locking down on a single feed format, even if today it seems like a great option, it might tomorrow show some problems. The metafeeds design allows this upgradability.

User has not chosen to be hosted publicly
User has chosen not to be hosted publicly
User has chosen not to be hosted publicly
User has chosen not to be hosted publicly
User has chosen not to be hosted publicly
User has chosen not to be hosted publicly
@Dominic %e19iJn6Ib93HSeHeIhq5Nfr5ig4vdHeoMKepSn/Gu2Q=.sha256

hmm, looked at nostr a bit, looks like they have something equivalent to legacy replication, which is fine for a small scale but would not replace twitter.... where as I think that ebt/request-skipping could do that

@Dominic %CvkxgYT9Mg2HBKjGiFdXbOYkvcJF6LS/+bQK2jHoqDA=.sha256

my recollection of bencode is that it's basically an awkward json...
https://en.wikipedia.org/wiki/Bencode#Encoding_algorithm

Hmm, it's length delimited for strings but the length is base 10 ascii, so this is the worse of both worlds. lists and dictionaries are not length delimited, and it's less readable than JSON.

I'm quite surprised that you'd choose this, out of all the possible options

@Dominic %5c8KzYbykIsG7r+dYvE9rTF0D1ZitDJXa8RwAemDid4=.sha256

hmm, it doesn't have whitespace is probably it's one advantage

User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
Join Scuttlebutt now