Next episode of my #dev-diary for #ngipointer. E01 about private messages and (re-)indexing was here.
The theme for this one is EBT, live streaming, muxrpc and how abstractions can haunt you. Since my work mostly revolves around #go-ssb, this will be quite #golang specific but I will try to make it entertaining for everyone.
Recapping the current state of go-ssb always strains me. There is a lot of If I wanted it would do X already nested inside hidden gotchas. Or simply areas where I hoped to get more help from others or just underestimated the involved complexities myself and couldn't get it done.
One central area for ssb is what we call live streaming. This usually means, forwarding messages as they arrive. You can do
createHistoryStream --id @theFeed --live (or the JS version of it) and it would keep printing messages of that feed for forever. Network outages would introduce a delay but once it recovers, new messages are displayed as soon as the local bot verified and ingested them. The databases also support this concept, which means that you can open a query for a thread or gathering and new updates trickle in as they permeated the network. Building reactive things with this is very neat (iff the UI is build so that it doesn't flicker but let's not go there here and now).
Now, how does this relate to go-ssb? Well, as you might have guessed it doesn't do it live. Conceptually the flumedb-port (margaret) supports live-queries but the legacy replication doesn't use it. Which means that it has to fire-and-forget the replication calls over and over, essentially making it a polling solution. Mostly because doing it live introduces a hefty overhead to juggle 15-20k individual queries, one for each feed.
About 11 month ago, when I was still working for planetary, I tried to re-write this central piece of infrastructure and tried to come up with hacks so that the overhead is smaller. I also wrote a new kind of test-suite for multiple sbots to connect and let messages trickle across different network setups like 1 to N fan-out scenarios, making sure that messages appear before a certain timeout.
The muxrpc re-write sadly turned into a disaster and it's name (SunkenSource) shall never be spoken of again. It was the leakiest and most unstable code I ever produced. Deadlines hamper with code quality a lot.
Now, a year later, I'm tasked with implementing EBT for NGI. And one central thing in EBT, apart from reducing do you have newer then X? overhead is that it's all live. So.. let's see what I came up with this time in the next post.