You are reading content from Scuttlebutt
@andrestaltz %R8gkAlWIdegMquiUmdGHhQyOw+7zP8mIj3nLROWCncU=.sha256
Re: %HqvC29wmr

Here we go again! After releasing rooms 2 and updating Manyverse a couple of times, I began a new task in ssb-ngi-pointer: an important refactor/rewrite of replication-related modules before we add support for partial replication.

Arj and I call it "The Bermuda Triangle": ssb-friends, ssb-replicate, ssb-ebt. These three modules are very important to SSB in Node.js, yet they are deeply intertwined with each other to a degree that is maddening. I made a diagram of which of these modules calls the other module, and it looks like this:

btriangle.png

We're about to build replication of metafeeds and its index feeds, so we have to add logic to some of these modules, and it would be horrible to just cram in more code into this mess. We need to be able to write code and be confident that it'll do the right thing.

It doesn't take an expert to see that something is wrong when these two files exist: ssb-friends/glue/replicate.js and ssb-ebt/legacy.js.

So I took a deep breath and started refactoring these modules. It started by modernizing the tests, making sure they run in GitHub CI for Node.js 12, 14, 16 and use easy-to-understand async primitives like async-await, not continuables. This was important because when refactoring you don't want to break existing logic. There was a ton of tests to update and fix. However, mistakes happen, sorry @Mix, I'll get to the bottom of that soon.

ssb-replicate is a little odd. It's legacy now that most/all peers have EBT, but you're still supposed to install ssb-replicate, configure config.replicate.legacy = false just to get an empty shell which ssb-ebt will monkey-patch. That sounds very unnecessary to me, and the obvious conclusion is that I'd like to deprecate ssb-replicate, in the sense of signalling on its README that you shouldn't use it anymore. It should still function as an npm package, just in case someone out there depends on it existing. Then, ssb-ebt needs to be changed to not have monkey patching. I started that already: https://github.com/ssbc/ssb-ebt/pull/45 see also https://github.com/ssbc/ssb-friends/pull/67

The basic plan is that I'll make ssb-ebt and ssb-friends become "passive" modules. They don't trigger or orchestrate replication, they just provide low-level APIs. Then, a new module ssb-replication-scheduler will use ssb-friends and ssb-ebt and be the "proactive" module responsible for causing replication.

In the process, we bumped into bugs left and right, which is kind of good because we don't want bugs, but also kind of scary to see bugs in something you thought was solid.

Also, as I test ssb-ebt (without ssb-replicate) in Manyverse I have a big doubt. It seems to replicate less content than ssb-replicate, but which one is correct? Does ssb-replicate fetch too much content (e.g. from blocked peers, or peers beyond the hops range), or does ssb-ebt fetch too little content (e.g. not fetching something that it should fetch)? It's very hard to discover the answer in large-scale deployments, due to the p2p nature. We hope netsim by @cblgh will help us answer these questions.

I joked in our internal chat that I should put "Proficient in Refactoring Dominic's code" in my CV.

User has chosen not to be hosted publicly
@andrestaltz %NVSwsyMmbSGxWanaOQx8sLquot1bZgp+0pKYKgu32rY=.sha256

@punkmonk Yep!

User has chosen not to be hosted publicly
Join Scuttlebutt now