dev diary - muxrpc

I've been working on rewriting muxrpc the last few days. This has needed to happen because it doesn't have back pressure... and streaming async systems need back pressure. I believe this is a cause a number of performance problems - currently if you do say, call createHistoryStream for an entire feed it will load all the messages and write them to the network without waiting for any kind of backpressure (and in initial sync, this will happen thousands of time in a short period)

When I originally started working on ssb I tried to make multiplexing that had back pressure, with pull-streams, but it was too hard... so I decided to come back to it later. In the meantime, used some hacks to get around the problem (such as for the UI using limit and instead of reading one stream, read many streams in "pages"

Since the muxrpc didn't use backpressure, it didn't use pull-streams either. It used something I called "weird-streams"... Yes, as a sign that was a duct tape solution.

A few months ago I had a discussion with someone calling themselves "SlugFilter" in the comments of my pull-stream blog post that made me realize a case I had missed with pull-steams (and not needed, anyway)

But looking at weird-streams again made me think I should adapt that idea to support back-pressure... and I did and I called it push-stream (I also remember @creationix working on a model like this too - he also independently pretty much came up with the same api as pull-streams (and used it in js-git) I remember he also explored a object oriented model but wasn't happier with it than closures.

Anyway, push-streams isn't quite as code minimal as pull-streams, but is probably easier to understand, also it's looking like they use less memory. It is much much simpler than any flavor of node streams, and in a microbenchmark, 3-4x faster than pull-streams, which is 6x faster than node streams (please remember this is a _micro_benchmark about setting up the pipeline, in a real world case, there might be so much other stuff happening that the stream overhead isn't relavant. That said, node streams are very bloated!)

Okay - so push-mux is not gonna start out with full back pressure, the initial goal will be to reflect tcp back pressure into the substreams. This will make syncing the whole database not load the whole database into memory faster than it can be written to the network. I'll measure this effect on perf and see how it goes!

yay, excited to see this important low-level part of the stack seeing some love.

i tried a few times to re-write muxrpc when i was writing vas, there's some juicy bits in there!

Cool, this happens to be a lot related to stuff I am working on, or have worked in the past. I'm currently getting muxrpc to work with web workers, and in the past I've been working with various "push stream" libraries.

@dominic when it comes to performance you probably want to get some ideas from most.js, the fastest (by far) push-semantics stream library. It's sort of like RxJS, but faster and meaner. It's also not "multiple destinations by default", which you may encounter mentioned as "cold observables" elsewhere (see also this blog post of mine). So a pull-stream API built on top of push-stream is literally possible (if I understood the constraints and requirements correctly). See this comment in Cycle.js development where I propose an idea like that.

Recently I had a big realization that a pull is just two pushes. E.g. even something as raw as the pull Math.random() could be split into two pushes: one push of null content for the request, and one push with a number as a response. Perhaps that sounds obvious, after all it's just message passing from A to B and back. But if push can model everything, than a push foundation (with pull built on top) sounds promising. This became even more relevant to me as I was working with Web Workers, that only support pushing a message (from either side). It hints that the lowest level of communication is a push API.

No closures are necessary (which are related to memory leaks and not optimized by js engines)

I know this is the common wisdom, related to hidden classes in V8, but I heard this is changing, and would help the implementation of many push-semantics stream libraries. Information about this is scarce, but here is one insightful comment by a person who wrote one of these libraries.

@andrestaltz most is a reactive programming library for UI development, it may well be execellent at that, but for building ssb we need a systems programming library. The important concern difference is that systems programming is all about back pressure (slowing down at the right times) and handling errors.

To make a fast stream library, well it's like making any other fast library. don't do too much. don't keep the event listeners in an array and then iterate over that array. Don't create new objects each run. The simplest cases in pull-streams are very fast because you just call one function. (rather, two because they call you and you call back)

In push-stream, a pipeline is just a double linked list, .write() calls flow from source to sink and until something has .pause==true, when that thing unpauses it calls .resume along the list in the other direction. The only state that changes on a stream in operation is pause (if back pressure is happening)

I'm not planning on rewriting all the pull-streams as push-streams but any good streaming library is easy to write adapters for...

@robert this is different to that. That approach was possible in that context because peer connections were fully random and frequent (every second, iirc) on the other hand we try to use long lived connections (to avoid handshake overheads)

hmm, not sure. I never read a well found "theory of streams" (which is maybe why stream libraries in different languages have their own problems) I can direct you to my own writing on it though:

• http://dominictarr.com/post/145135293917/history-of-streams
• http://dominictarr.com/post/149248845122/pull-streams-pull-streams-are-a-very-simple

I did read stuff but most of my knowledge comes from working with streams in node.js.

Oh also, it would be damn easy to apply the push-stream api to another language. The pattern doesn't require callbacks just basic OO.

@dominic I wrote this for different purposes but it shares a lot in common with push-streams: https://github.com/staltz/something-something just FYI

Thanks @Dominic for being bloody clever and taking the time to add the documentation layer. I feel lucky to be learning along with you

@dominic I think I experienced this lack-of-backpressure problem now with muxrpc on mobile. It may be important to solve in order to get better performance.

Basically I have the setting main thread <--> worker thread <--> bg process, and I do one pull for createFeedStream on the main thread and through muxrpc to worker thread it reaches muxrpc to bg process and after that I get a huge amount of ssb messages coming back to main thread. I'd expect one pull to give just one ssb message.

I believe some of this is choking the mobile app, for instance when a pull-to-refresh happens, the previous frenetic flow of ssb messages is still ongoing and the bg process hasn't yet handled the latest "pull", it's just busy dumping data to an older callback that is no longer used.

@andrestaltz currently you need to use pull-more like this: https://github.com/dominictarr/patchapp-threads/blob/master/public.js#L31-L42 that will only read a fixed number of messages and not request more until you've used them.

I'm making good progress with push-mux (this morning I just implemented a first go at per stream flow control and it's working!)

Great! I didn't know about pull-more, and even the readme for push-mux is good material to be aware of. Thanks

Okay, cycling back to this after rewriting epidemic-broadcast-trees and I now again have a working implementation of muxrpc, but now internally using push-streams. I had to change some of the muxrpc tests, so I'll consider this a breaking change, but scuttlebot and secret-stack tests still pass without changes so it probably doesn't actually break anything!

I have linked it into scuttlebot and messages have been synced via it!

Open question: now peers can "allocate credit" to on a per-stream basis. Currently there is just a fixed amount that gets renewed when it's exausted, but it's still better than the old way without back pressure. There should probably be something that estimates the rate of flow maximizes flow while still allowing back pressure to occur if possible.

Okay, I just published muxrpc@7!

There is also a very exciting opportunity that this unlocks: what is called relay-p2p
in my TechTree. On the one hand, this would enable bootstrapping full p2p via relays through pubs (and also the ability to fallback to tunnels through pubs). And on the other hand, this would allow ssb to provide direct p2p connections to other applications that use ssb identities. For example, you could have a realtime chat protocol strapped on the side (or other analogous things, like real time collab, games, etc!)

I'll probably work on private groups next (after this current replication/sync proposal) since I think there is more general excitement about that, but also I think this relay-p2p would be a good chunky task for someone, that really adds something exciting and is challenging, yet does not intersect with the entire stack like private-groups would.

Great news! @dominic does muxrpc@7 have breaking changes? (I'd guess so, according to semver) I couldn't see from the readme or the repo what are those, if any.

I'm using muxrpc also with "Web Workers" on mobile and I might as well update it.

Super exciting times :D.

When you're using ssb-chess to have a real time chat during games (there's a chat box to the left of games), it is something you notice sometimes (the few seconds latency between messages, depending on whether you're connected to the same pub at the other person or whatever, etc - varies a wee bit sometimes.)

@nichoth it was something that was mainly difficult in the context of pull-streams. Check out the articles linked from push-mux readme.

@andrestaltz yes, but only tiny. Previously muxrpc@6 had a thing where you could call rpc.close() and it would finish what it was doing and then close ("graceful close"). Now it just aborts anything currently underway. The "graceful close" was needed by scuttlbot tests at some easier point... but this was more a source of complexity, so removed.

I didn't actually need to change the scuttlebot tests, so it's unlikely it breaks anything, but I made it a major change because it was a big rewrite internally and this means users can opt into version 7 as they choose to.

@nichoth "difficult" is very subjective, so it's hard to say really. And my point of view is probably not a good guide in general because I have a years of experience writing streams. I do remember watching peoples faces as they worked through the pull-streams workshop. The functions returned by functions etc is very powerful but sometimes difficult. I think in particular, closures are not as explicit about what the state is at a given instant.
The state is represented as a bunch of local variables, for example in pull-defer/source. With push-streams, the state is a little bit more explicit.

But more importantly, the code is flatter. You just check if this.sink.paused==true
and try not to call sink.write in that situation.

Also, push-stream does way better in benchmarks! it's significantly lighter that pull-stream (but more lines of code...)

@Ben are you prepared to implement relay-p2p?

I'd start by creating a sbot plugin with a duplex stream, which you run on pubs
and peers. A peer could then call that with the friend id they want to connect to,
and then if the pub receiving that call is connected to that peer, they'd tunnel back to
the peer and pipe the caller's stream to that.

There might also need to be some way to advertise which pubs you are connected to,
and thus reachable at... but a simple version would probably work without that.

I have fixed two problems caused by muxrpc@7

One was that push-mux wasn't falling back correctly when it's remote didn't support credit based control flow, this is fixed in push-mux@1.0.3

The other was that muxrpc@6 was hanging open if you don't use rpc.close(true) (close immediately, destroy) if you just use rpc.close() it would wait for the incoming credit control stream to end properly (which is also fixed in push-mux@1.0.3) except it turned out that muxrpc@6 also had a bug that didn't correctly end streams that errored. So this also required a patch to packet-stream@2.0.3

reinstalling muxrpc@6 will get the patched packet-stream, but if not it may hang open.
and you'll see a message in your standard error:
no .read for stream: -1 dropped: { req: 1, stream: true, end: true, value: true, length: 4, type: 2 }

In particular this effects git-ssb, however reinstalling it should fix everything.

please confirm this fix @matt
@cryptix!

@Dominic

please confirm this fix

doesn't look like it worked

Patchwork is still hanging on startup. I checked to make sure push-mux@1.0.3 is running, and it was.

Don't have time now, but will investigate this further later, and see if it is the same issue as before.

I think I got a working sbot with muxrpc@7 install but came across this sbot crash using it with git-ssb:

/home/user/scuttlebot/node_modules/push-mux/index.js:217
      req: this.controlStream.id, stream: true, end: false,
                              ^

TypeError: Cannot read property 'id' of null
    at Mux._credit (/home/user/scuttlebot/node_modules/push-mux/index.js:217:31)
    at Sub._preread (/home/user/scuttlebot/node_modules/push-mux/sub.js:54:15)
    at Sub.DuplexStream._write (/home/user/scuttlebot/node_modules/push-mux/stream.js:31:10)
    at writeDataToStream (/home/user/scuttlebot/node_modules/push-mux/index.js:83:9)
    at Mux.write (/home/user/scuttlebot/node_modules/push-mux/index.js:161:15)
    at next (/home/user/scuttlebot/node_modules/push-stream-to-pull-stream/sink.js:30:21)
    at /home/user/scuttlebot/node_modules/pull-goodbye/node_modules/pull-stream/throughs/filter.js:17:11
    at Object.cb (/home/user/scuttlebot/node_modules/packet-stream-codec/index.js:111:11)
    at drain (/home/user/scuttlebot/node_modules/pull-reader/index.js:39:14)
    at more (/home/user/scuttlebot/node_modules/pull-reader/index.js:51:13)

but I also wasn't able yet to build it myself with updated packages.

hey @dominic! We tried to upgrade to mux7 for our release of #talenet alpha6 but had do downgrade to 6 for now. @baldo has a performance degradation after some reloads which we need to look in deeper.

We are already using mux7 again on our development branch but we needed to make a custom sbot-muxrpc7 branch to pull everything up. This commit is the most relevant patch, I think and I'd like to have it in mainline scuttlebot.

I also got git-ssb working again after updating. Updating patchfoo seems like it might be a doozy because if the custom packages that cel made. Will also see if I can get patchbay and patchless up and running. I found two new sbot crashers though. This one and this one:

Exception thrown by PacketStream substream end handler TypeError: this.read is not a function
    at PacketStreamSubstream.destroy (/home/user/patchfoo/node_modules/packet-stream/index.js:286:12)
    at PacketStream.destroy (/home/user/patchfoo/node_modules/packet-stream/index.js:81:24)
    at PacketStream.write (/home/user/patchfoo/node_modules/packet-stream/index.js:133:41)
    at source (/home/user/patchfoo/node_modules/muxrpc/pull-weird.js:43:24)
    at /home/user/patchfoo/node_modules/pull-goodbye/endable.js:6:7
    at /home/user/patchfoo/node_modules/pull-through/index.js:31:16
    at /home/user/patchfoo/node_modules/pull-through/index.js:31:16
    at source (/home/user/patchfoo/node_modules/pull-pair/index.js:21:7)
    at /home/user/patchfoo/node_modules/pull-cat/index.js:7:11
    at Array.forEach (<anonymous>)

@cryptix thanks!
those are both look like easy fixes. (but both in packet-stream and push-mux)

If people building frontends on sbot make some sort of automatic tests (anything really, even just "did the first screen load", as long as I can npm test then I'll run those tests before publishing new sbot stuff). This passed all the tests that I had, so I published it.

Okay, I think these two issues should be fixed in packet-stream@2.0.4 and push-mux@1.0.4

I'm really sorry about this, but don't know where else to put it. Wanted people to know that to get Ticktack standing again I followed Matt's lead on pinning some packages. Noted it in the README:

There are some issues with some lower level scuttlebutt deps (scuttlebot, muxrpc?). For the moment we've pinned these deps:

"scuttlebot": "10.4.10",
"secret-stack": "4.0.1", (added in order to stop version of muxrpc drifting up. can remove later)
"ssb-client": "4.5.2", (added in order to stop version of muxrpc drifting up. can remove later)

@dominic updated those two and don't see any printed errors yet but I now see what @baldo also noticed quickly. After some time sbot needs a really long time to handle queries. Up to the point where the blobs server (I also patched ssb-ws to use muxrpc@7) needs 10 seconds to serve an image. I've produced this with our application now semi-reliably (enough reloads) and think I've also seen it with git ssb.

yet but I now see what @baldo also noticed quickly

what was that?

Up to the point where the blobs server (I also patched ssb-ws to use muxrpc@7) needs 10 seconds to serve an image

blobs served over muxrpc? or http?

considering making a breaking change to push-stream, not fully convinced yet, though. But doing so would be relatively easy at this stage (there are only 3 push-stream modules) https://github.com/push-stream/push-stream/issues/3

This would also make it even closer to reactive-streams which seem to only be implemented in java... maybe that has implications for android though?

I hereby pronounce ssb-ebt@4 sufficiently stable to try on your local setup. There are still some more things I want to do on it, so maybe don't go shipping it in builds, but is feeling pretty stable. I'm gonna update my pub to use it too.

This would also make it even closer to reactive-streams which seem to only be implemented in java... maybe that has implications for android though?

I don't think I see any use case for reactive-streams (which are meant for busy servers) on Android, at least not in my current app. My custom use of muxrpc@6 doesn't cross any network boundary, it's all local (in the same process) and Java is involved just to bridge the gap between JS main thread and JS worker thread.

I haven't tried muxrpc@7 yet, and I'm not sure how the whole credit flow and pause/resume will affect the workers thread architecture. request(bytes) seems like a good idea though. I'd say take your time to think this through, because it would be better to migrate just once to the new muxrpc/push-stream arch.

@andrestaltz I'm mainly thinking we'd use it in the bit that adapts the javascript part to the java part. reactive-streams are a lot more nodey (because they are non blocking) than the classic java stuff, I just mean use reactive-streams on the java side, then it would be easy to have a javascript adapter to that. Once it's in your app, you could use pull-streams or callbags or whatever.

Update: by eyeballing the ebt logs (set ebt.logging = true in your config) I noticed a bug then added tests and fixed it. update to epidemic-broadcast-trees@5.0.3

I ended up spending all day working on muxrpc after I tried doing an initial replication over muxrpc@7 & ebt@4 and realized how badly I screwed up. I had the basic logic for the cbfc sorted out, but didn't have test coverage for actually using byte credits... so the server was sending 64k then waiting for more credit... but the client was only counting messages... but it takes a lot longer to count to 65365 messages than bytes, so the server was pausing, but the client wasn't asking for more! It took all day to fix because some refactoring was necessary... it's past midnight now so I'm gonna leave it till tomorrow to publish anything...

yet but I now see what @baldo also noticed quickly

what was that?

I've been developing #talenet all day, so a lot of restarting and initial queries. Over the time it grinds down the sbot nearly to a halt. Really slow responding until the client connection times out. I need to restart it then. I even tried sbot gossip.disable global to reduce it's workload but it still occurs.

blobs served over muxrpc? or http?

served over ssb-ws

@cryptix sorry about this. The solution is just to go back to muxrpc@6 for now. I published patch versions of secret-stack and ssb-client that downgrade muxrpc, so reinstalling should do that.

yup, already did that for #talenet. Feel free to ping me once you want us to try again.