Glad that you were able to run the debug app from your environment! I'll update the readme with some of the steps you used.

And feel free to ask if you have any question

@piet do you want feedback in your diary on in a parallel thread ?

TOO LATE

thoughts so far :

1. Prioritisation of peers
   • how about numeric values? (Low) 1,2,3, ... 9? (High), -1 === 'BLOCK'
   • avoids magic strings ... mind you redux is all about that magic!
2. Where does this fit in the stack?
   • I'm new to this, so am surprised when init takes some connectToPeer? is this for testing
   • does this manage things automatically for me? The text "that manages making connections to other sbots" makes me think it foes, but the features suggest this is a providing accessible methods for managing connections myself ???
3. Does this follow the 'instantiation pattern?
   • start and instance var manager = require('sbot-connection-manager')(opts)
   • use the instance: manager.startInitialSync()
4. I've become a real big fan of methods nest in 'domains', as I think this makes it easier to clearly group concerns, and provide shorter and more legible names e.g. :
   • manager.initialSync.start(), manager.initialSync.stop()
   • manager.peers.setPriority, manager.peers.add, manager.peers.obs
   • manager.connections.start(), manager.connections.seLifetime, manager.connections.peerConnected

@piet in terms of "where does this fit in the stack" can you show me 20-40 lines of using this... it will help me as the reader know where this is useful, and whether I need to be concerned with it.

• e.g. is this is you're rolling your own sbot?
• is this for if I'm running an embedded sbot but want to give devs of clients users realtime granular control over replication ?
• will this just be a nice module hidden in the stack that only backend gurus will reall touch.
• is this an sbot plugin?

@Piet I finally got to read the readme carefully. Looks good! Great job at readme-first development. Here are a few comments, but even if the comments aren't considered, you're already on a good track.

Set maximum number of peers to connect to.

It could be a problem that your sbot never connects to your very close friend because it's already connected to N other (not so close) friends. While in principle it sounds reasonable to limit the number of connected peers, we need to consider the corner cases. I'd say for now, use a very large max number, and then later we could discuss about this problem. Maybe Dominic finds some clever distributed algorithm in some paper, for this.

Provides a stream of peers with errors. Useful for another module to decide which peers should be forgotten and which could be retried occasionally with low priority.

This is very good! To me it's a new idea, I think it enables a lot of modularity.

Which led me to realize that perhaps this module could be split into a few modules. It's heading towards becoming a large and important module. I think we could identify an unopinionated core, made of connection tactics:

• Encapsulate the peers data structure (add/remove peer, set peer priority)
• Connect individually to peers (like current gossip plugin sbot.gossip.connect(peer))

Then have some other module made of connection strategies:

• With the API "startConnecting()" without specifying which peers
• With the API "startInitialSync()"

Hopefully modularizing like this will help to avoid making another monolith gossip plugin. Then it would be easy to maintain a simple core "connection plugin" and on top of that different plugins (each with different opinions) for managing multiple connections.

There are currently limits on the number of connections at any one time, but they are separated by type. we can have a different limit for global (remote internet connections) and local network connections. Sometimes a particular protocol is more expensive than others - for example, I've heard reports that webrtc has big problems when you get more than 4 connections (which is a big problem if you are trying to implement a DHT, but not a big problem for ssb)

A big part of what gaurantees the eventual consistency of a gossip network is the randomness - because if there is some chance that you'll connect to a given peer, you know that you will eventually connect.

@piet sorry just noticed this now.

Should some randomness actually be built into this module?

as long as there is some randomness you'll get eventuallity. for example, have several connections to the "best" peers, but then have another one that randomly walks around the others.

hey @piet! I saw the PR and am really happy to see this happening since I totally missed this was going on!

re questions

what isLocal?

If I remember the code right, this is about the local network case, i.e. connections coming from local UDP broadcasts.

why do Peers exist as distinct from Routes
peer is mainly useful for blanket-banning any connections to a route

you definitly want to talk with @arj about this. He currently tris to add graph-distance checks to incomming connections. See https://github.com/ssbc/scuttlebot/pull/530 if you haven't.

could have a priority list by connection type e.g. isLocal, bluetooth, onion, tcp
how do we ban people from connecting to us?

this sounds like the scopes and connections stuff we have been brewing recently. A priority list for this would be really nice!

Prefers peers that haven't had connection errors over ones that have.
Prefers peers that haven't been connected to in the longest amount of time.

I really like this thinking but from my experiences and experiements with ssb-pubmon, I know that pubs can be reeeeally flacky! Some of them just time-out on you even the tcp listener is up and the next day they happily while others are down for ages. What I'm saying is +1 please go forward but try to keep recently working connections higher in the prio than simply pushing back because it didn't work once. Maybe some high cut-off before it's regarded as offline or some kind of tryAgain: [key,pair,list] in the config?

lots of &

hey @cryptix the question "what is Local" was one I raised. We know what it is in the code right now, but I was trying to dig for what it means or should mean. Is it about whether something should be prioritised? If it is, then shouldn't it be a priority property? I'm thinking towards a future where there could be several ways for a peer to be "local" wifi, bluetooth, a direct webrtc connection?

With the list for how to reconnect, that is literally the simplest stupedest algorithm that piet could implement. I suggested starting stupid because it's quick and can be improved.
The focus of the conversation we were having was "get it all connected in a simple way". Worth knowing that Piet's style is to go kind of wide and thorough, so the connection-manager is well built, but I think it's time to plug it in before going wider again!

I suggested starting stupid because it's quick and can be improved.

I see, it's just that I fear running a peer with

Prefers peers that haven't been connected to in the longest amount of time.

will result in endless attempts of all the offline-junk, that is most of the type:pub messages. I tend to configure my pub for dial-in only anyway but the default might become iffy for some installations.. Than again, I have no idea who often pubs are updated once they were created... more motivation to bring pubmon to a usable state... :)

Maybe it's more a question of what we relase to whom? I'd be definitly willing to test this branch, maybe from towmorrow or later today.

That's the only thought I really wanted to raise with this.

Re isLocal I'm sorry I re-explained something you already knew. I just wanted to give some insights and see if we are on the same page what and why this was (invites, skip friend graph on local network). Going foward, I totally agree - see my involvment in scopes and connections.

let me make a bit more progress!

sure, don't worry, @piet! I just felt like ing around a new sheduler implementation.

Let me know when the time is right!

Can you link me to your scopes and connections work? Haven't seen that.

Sure! I just wrote a high-level description for the readme of ssb-config.

The this spans mutliserver (merged PR), secret-stack (unmerged PR) and ssb-config (unmerged PR) before we can update scuttlebot. I polished the multiserver-config-scope branch and opened a WIP/tracking PR as sbot#536.

cc @arj @andrestaltz fyi :)

@Piet Thanks for reporting that error. It should be fixed by changing ssb-config to 2.2.1 in src/nodejs-project/package.json. I'll commit that to master branch soon

should this replace existing gossip or just be a separate plugin?

In my opinion, yes, replace.

@piet @andrestaltz after the initial sync, most syncs arn't really that big, and ebt already skips most of the heavyness of legacy replication (because it avoids doing thousands of createHistoryStream requests)

I'd ask that we just relax the requirement for throttling to instead be just on/off, or to throttle by varying the number of peers to connect with at any given time.

When we get bpmux we'll be able to make ebt streams go slow or fast, and adding an ad hoc back pressure mechanism to ebt right now will expand the scope of this current work unreasonably. I think it's better to leave it and solve it in general with bpmux.

Yes it's good to review what should be done because some time has passed since the quest was specified. Like some of those are done already, through different means:

Should: implement some indicator in the app showing that replication is currently happening
... interaction responsiveness ... got better when the app is connected to pubs and has just been opened

I also think we should relax the requirement for throttling, but it seems we really need on/off for each connectivity mode. I don't agree with Dominic on this:

after the initial sync, most syncs arn't really that big

Because it really depends how often you open the app, and how good your internet connection is. If you open it every day with fast internet, it's okay. If you open it every month with slow internet, it's as bad as initial syncs are. In places without internet, the use case for the app will be syncing a lot of content every once in a while (every week?) through a slow internet connection or through Bluetooth peer-to-peer.

We also need to make the experience of initial sync better, because currently it's "wait for 8 hours for a lot of content to come in", and people may not even be that sure whether they want to commit time and bandwidth to download all that content and only after that start to explore the Scuttleverse. There must be a way of just sipping some content to begin with.

That's why this quest is about prioritising as well: can we just fetch hops=1 data to begin with, and if the user is interacting with the app, stop the replication and let them interact with it. When the app goes to the background, resume replication, etc. Once all the hops=1 feeds are replicated, proceed to replicate hops=2 feeds, still obeying the same pausing behavior if the user is actively interacting with the app. Etc until we replicate hops=N where N is in the config. Also, the user should be able to manually pause/resume the replication (it currently appears as a notification on Android, it's easy to add some buttons on that).

Some of these features are related to ssb-friends, some related to the replicate plugin, some related to EBT, etc. Makes it hard to separate the work.

But most of these concerns are covered by having a sane re-implementation of the gossip plugin, which I think is the bulk of this quest. A well built connection manager that is agnostic to connectivity mode, and makes it easy to pause/resume any connectivity mode, etc, will help us get to the goals we need. One example currently the gossip plugin parses addresses with its own functions or with ssb-ref, and often doesn't support anything other than ipv4 or ipv6. We would need to delegate the parsing to multiserver, which automatically tries all the parsing functions available and returns the first one that worked.

I'm not emphasizing the particulars here, whether it's throttling or back-pressure and which module are we touching. To clarify what "prioritise" means, it's more important that we can solve:

• App developers can easily tweak the syncing (gossip/ebt/friends/replicate) behavior
• Time to interact with content in the Scuttleverse after installing the app is reduced to just a few seconds (max 20 seconds?)
• If the user wants to interact with the Scuttleverse while syncing is happening, syncing stops and we dedicate CPU to support UI interactions

@Piet looking good! My feedback is coming from a "blurred" vision of all these modules (just giving some context, that I don't have a perfect assessment of the situation), so one question I have is: how are you going to stop ssb-friends from calling replicate? These lines https://github.com/ssbc/ssb-friends/blob/173bf283c5a58a266026c813f7df62cd9f9605a4/index.js#L115-L123

In my opinion, gossip & replicate & ssb-friends & ssb-ebt is a mess, and would hope you could untangle some of that mess. E.g. it would be good if ssb-friends was passive: it only answers questions, it doesn't proactively do something. Because that's its actual promise: "the logic around who to replicate or not", but those lines are doing much more than just being the logic around who to replicate, they are also actively asking for replication.

I still want to update the https://github.com/ssbc/stack-diagram so that it reflects this entanglement between gossip & replicate & ssb-friends & ssb-ebt so we get a better understanding of it, but as inspiration (not request), I wish it would be:

Layer 1:

• connection-manager (stores a list of connectable peers, passively pauses/resumes, proactively chooses who to connect to)
• ssb-friends rewritten a bit (passively answers questions like isFollowing & isBlocking & hops streams)
• ssb-replicate-simple (a rewrite of the current sbot/plugins/replicate)
• ssb-replicate-ebt (I just noticed ssb-ebt and sbot/plugins/replicate are basically mutually exclusive! the former fallbacks to the latter)

Layer 2:

• replication-manager (roughly like you sketched, but it allows choosing between ssb-replicate-simple or ssb-replicate-ebt, and does this logic which is also configurable whether we first loop over hops=1 then hops=2, or whether we mix all hops together)

The reason I'm thinking in layers is that it would solve some of the entanglement between these modules. In principle, Layer 1 modules should not assume the existence of other modules in Layer 1, they only have access to Layer 0 modules (e.g. secure-scuttlebutt, multiserver, ssb-config, etc). Then, Layer 2 modules assume the existence of Layer 1 modules and mix and match their APIs. In other words, 1 is local tactics, 2 is strategic. Then UI apps can just communicate with Layer 2 modules and do operations like: configure the replication strategy, start/stop, query the progress. I don't think much more is needed in the API of a high level module like that.

Anyway, probably more feedback than you needed. I don't know how much of all of this is doable in this quest, but the more clear our goal is, the better we can know whether what we're doing is valuable and useful in the long term. It seems like a good action plan would be to start rewriting Layer 1 modules first.

@piet asked (way back at the start of the thread, but I just noticed now, sorry):

Doesn't the ebt tree get invalidated every time we break connections and then connect to a new peer? I'm imagining the "tree" getting constantly broken and remade in different ways.

yes. ebt is really the optimization for realtime messages (and request skipping is the optimization for connections). So it only affects connections that last a long time with respect to message frequency. But that's helpful because the protocol doesn't really have a distinction between messages that are "realtime" or not.

Put another way, would ebt be more efficient overall if we all slowed down the rate that we cycle through our pub connections? Or am I fundamentally misunderstanding something?

yes, also this helps request skipping too. as described in the draft paper (oh, I did a bunch more on this a few weeks ago) RS works best if you have a fixed number of peers - say 5-10. If you connect to random 5 random peers, the chance that the network will be fully connected is very high.

Oh, now I remember, I started some work on sampling how many friends you needed to replicate with to statistically get a high probability of getting everything you'll want. (say, nearly certain to get all 1-hops, and probably most 2-hops)

This sounds good to me as well. @keks also raised the point that it would be very nice to be able to use ssb-ebt in our go-ssb as an out of process plugin.

For this to succeed in general we need clear boundaries between plugins and the supplying sbot, APIs have to go through muxrpc and not depend on jsbots internals. This layering you described seems like a very good starting point for this untanglement work.

I don’t want to add more cooks to this then necessary but let me know when there is anything. I hope to get startet on OOP about two weeks from now, finishing off basic scuttle-Shell around mid next week.

@piet can you share some context on what it's not letting you do and what api might let you do it?

@piet right, that api exists to support the various ways that things might choose to ask things to be replicated. for example: replicate friends, but also anyone who's local. If you want to intercept all calls, you could do that via hook.

What are you using that for, are you using it to control the number of feeds currently being replicated?

@piet are you doing anything special to select those feeds? (such as checking hop distance?)

One idea is instead of having request be a boolean, it could be a number, and at any time you could choose to replicate some number under a threashold. say, the first 100 that are at least 1.5 away, then expand or contract that. (of course this will be very order dependent... but that can be delt with separately)

This could be added to epidemic-broadcast-trees fairly cleanly, without changing it's fundamental concepts - just checking that a threshold is met, instead of a boolean, and when it loops over peers, it would include a counter too.

If the peers are already in hops order (and I think they do come out of ssb-friends that way) then just implementing the count would be enough.

@piet what if instead of an array, you could just give each peer a priority number?

@piet okay,

currently to say replicate something, you do:

sbot.replicate.request(id, true) //give me id
sbot.replicate.request(id2, false) //I don't want this anymore

but what if it was:

sbot.replicate.request(id, 1) //2 or 0.5 or whatever
sbot.replicate.request(id, 0) //disable
sbot.replicate.threashold(1.5) //replicate anything with a number under 2

my motivation for the request method, is that there are a bunch of different things that might have reasons they want to replicate something, but those things don't need to know about each other - any plugin should be able to request a replication, we shouldn't maintain that in just one place.

Limited has a threshold, and when a peer you want to replicate is further behind than that, the manager goes into limited mode and only replicates a limited number of feeds at a time.

@piet I'm excited by this! At the moment, my bluetooth connection explodes after replicating for a minute or two and I suspect this might stop that from happening :).

@piet I don't know which of those it is, it's a suggestion (and a thought that occurred to me previously). The point is to see how you react, and thus extract more context about how you are thinking about the problem. That this suggestion has fallen somewhat flat actually tells me quite a lot - the way you were approaching the problem was quite different. Of course, you were approaching the problem the obvious way, to you but I don't know how that differs from my way, and neither do you. So I just suggest things and see how you react.

Another piece of essential context: ebt/replication is the heart of ssb, I want it to be really really good. If you can't do what you need with my api, there is two basic possibilities: either A) there is a better way to do the thing you are trying to do, that may be easy for me to point out, given my deeper understanding of the code, or B) I didn't anticipate your needs, and thus implemented the wrong interface, and so the code should change.

I wouldn't say it's a playful musing though, it's more like:

• here is a suggestion that would be straightforward to add to the ebt code, would it solve your problem?

and the answer seems to be "no, I actually have a different problem". I hadn't actually considered using the number of messages you expect to receive as the prioritization. And that leads me to more questions:

• what happens if someone new comes into replication scope, and they have a massive feedL (they are newly a foaf, does it skip updates from close friends to pull that down?)
• to know someone is far ahead, you'd need to receive the remote request, so you'd need to ask for that feed, but you'd need to ask for it in pull mode (rather than push mode), which isn't currently accessable but could be an easy PR to epidemic-broadcast-trees.

@piet if you feel I have missed something, please point that out!

I can't really answer a question like "do I understand your usecase?" we can only talk until we both feel like we understand each other. Also, just because I read something, doesn't mean I understood what you ment, I can only react by my interpretation of that. Please don't assume that something I say is automatically right just because I wrote that code.

The things I am suggesting are all things that I think would be simple to add, or necessary to make it work right.

On that note, I've just realized that you can't just flipflop replicate.request, because when you do replicate.request(id, false) it will send a "I don't want id signal" (-1) and that will screw with with request-skipping, because if you disconnect with the remote during the -1 state, they will remember that you don't want that feed and not mention it next time. (this may break eventual consistency - there are loads of tests in ebt to catch things like this)

Instead, we want to send a the signal that we do what that feed, but not right now! This is already sent if you are getting it faster from another peer. See the code in epidemic-broadcast-trees/v3.js (and v2.js was the old way to encode that). for example, epidemic-broadcast-trees/events.js timeout (line 390) checks if we are expecting messages on a feed but havn't got them, and cycles to another peer for that feed. it calls setNotes(peer, feed, seq, rx) to tell peer that we do (rx=true) or do not (rx=false) want to receive feed from them, but that we are up to seq.

To pause a feed, we just need to be able to set all peers to rx=false for a given feed.

@piet be my guest! then two people will understand it!

@piet yes that is how many tests it needs! Okay, how about this: use request for now, then we switch it to use ebt.pause(id, bool)?

Actually, what would the most convenient api look like?

it could be a batch pause({<id>: <bool>,...})?

anyway, this is why I suggested just handling replication as off/on and focusing on connections... since we definitely need that, but I think replication management will be more elegantly solved by giving muxrpc back pressure.

why two args?: hmm, I guess because if you don't want to replicate something, thats feels like a special case, a priority so low that you don't even replicate it. If you don't want to replicate it, the priority means nothing. This is just a style thing, though.

@piet of course, I didn't write that many tests, I generated them from a model. There is an essentially unbound number of possible orders things could happen in, so I test by sampling random orders (much easier than figuring out how to exhaustively generate all possibilities). I've found and fixed things that only happened 1/10,000ish times, which could be, happens to someone every few days, or happens to one person every few months. Of course, it would be very very hard to notice that something like that is even happening.

• Replication manager
• ssb-ebt branch
• ssb-friends branch

Looking good!

The app crashes within a few seconds of starting up and I don't see any error messages in adb.

Nothing in adb? That's odd. Which Android OS version (only 5.0 and above is supported).

How are you doing dev on node modules that you are using in the app?

One trick I use sometimes to hack things in is by editing nodejs-assets/nodejs-project/index.js (which after npm run build-nodejs-app should be one huge JS file) directly, and after that running cd android && ./gradlew clean && cd .. && react-native run-android. It's not as great as npm linking, but at least you get to prototype stuff without putting them in git or npm.

run $ react-native run android which always fails the first time

Fails why? My development process is usually (supposing I just git cloned):

• npm install
• Open Android Studio, open the Manyverse android project, press "Sync gradle" button, and wait for nodejs-mobile-react-native to fetch the NDK it needs (this step is needed only once per top-level npm install)
• npm run build-nodejs-app will npm install the backend project, build native modules, and noderify the index.js
• In a separate terminal, I run $(npm bin)/tsc --watch
• In yet another separate terminal, I run npm start -- --reset-cache so it runs the React Native packager
• Those two terminals stay continuously running in the background
• In my main terminal, I run react-native run-android, and it works (I don't need to run it twice)

Thanks @Piet for the PR, it seemed quite a good small amount of changes (meaning good modularization).

I tested it by recording the initial sync experience between phone and my computer on LAN. The app on the phone was installed from scratch, so it had an empty database.

When in the same LAN, the mobile sbot begins syncing with the computer's sbot (which is @andrestaltz), and after ~2MB of transfer, it's done. At this point, I think mobile sbot is not trying to replicate anyone else, because it is still not technically following the computer account. This means both before and after the PR, we have the same experience, because the mobile sbot is replicating only one feed. During this experience, we see the notification progress bar (and it updates smoothly every 1 second), and if we refresh the feed we immediately see updated content. We would also want to have this same experience when syncing many accounts (that is, we want notification progress bar and we want refreshing to show the newly updated content). Because the experience before and after the PR were the same, I don't need to share the video recordings of how that went. It was nearly identical.

What matters is what happens after the mobile sbot follows the computer sbot. After that, they are at hop=1 range, and the mobile sbot should begin asking for hops=2 feeds, and this is when (IMO) the replication limiter should kick in. I thought that due to the use of the replication limiter, we would see the notification progress bar, and we would immediately see new content upon refreshing the feed. To my surprise, though, the use of the replication limiter made no noticeable difference.

See this video recording of before and after the PR:

dat://compare-replication-manyverse.hashbase.io

How do you want to test this improves the responsiveness of the app?

Going forwards, I think you could do this initial sync test on your own, if you have a mobile phone. Just attempt to LAN sync with your computer's account like I did above. A success case will allow new content to flow in while still allowing the user to browse content, and clearly notifying the user of progress through the notification progress bar.

piet to write up learning about quest

• stack pits to know about
• accomplishments
• social dynamics (collaboration, quest dynamic)

@piet looking forward to this. I'm happy to support you to reflect and write that up.

I think @dominic and other back end devs will find your learnings really useful

Thanks @Piet for your half-a-year effort on this quest! The parts that went downhill were sad, but I'm glad we got to a good conclusion (essentially what you answered as the happiest path forwards is how we're concluding this, verbatim). I'll make the second payment in ETH as soon as I get your wallet address. I still want to do a proper retrospective that covers all the quests, and it touches topics such as video calls. On this quest in specific, it was very challenging (from the beginning) technically and coordinationally, because it involved legacy and complex code, communication with Dominic and me (we're both quite busy and hardly can do good guidance), known unknowns and unknown unknowns. I was expecting that the quest adventurer would be up for a very intense and independent adventure, but expectations are also the kind of thing that should be communicated a priori. All that said, this thread is now immutable and recorded forever on our butts, will serve future adventurers well, with 2 new modules, lots of technical discoveries, and data on project coordination. Initial sync and replication performance is probably going to be a never-ending source of problems, so this thread is definitely valuable.

That none of us really knew where the slowness was in the code until Andre did some profiling to test the replication throttling work

So what exactly is the slowness?

@arj I think he was referring to this thread: %3hli4+j...