SSBC grant proposal - Benchmark CI system

Performance of ssb is a key user experience problem. Also, performance determines how weak a device it can effectively run on, which is the same as saying poorer people can afford a suitable device. Theirfore, performance is essential to network growth.

In the past, we have not always kept track of performance and had some regressions. We've normally heard about problems from users complaining it's taken too long to load! This is not good enough.

This proposal is to build a benchmark CI system that will run every time a ssb-* module is updated. This means we'll know what the performance is at any time, and whether changes proposed improved something or not!

motivation

This is also intended to make working on the systems/backend side of ssb more accessible. Improving benchmark perf is a fun game that also involves learning how it all works. Currently, we have lots of people building front end applications on ssb, but I'm still doing most of the dev on the systems side. That is a point of centralization, we need to decentralize backend dev!

Boxes ticked: growth, learning, income, avoiding burnout

who will work on this

myself and @arj are gonna collaborate on this!
I've already started on some benchmarks bench-ssb and have already learnt a lot! We'll still need more benchmarks than this, and to display graphs etc, for the each run.

We've also discussed driving this through git-ssb, and npm-ssb!

@arj has full time work, so we'd do this part time over a 2 or three month period.

@arj one thing is that the performance of a run could depend on sub dependencies. for example, I found that ssb-friends was a bottleneck, but I didn't need to change the api to fix it, so it was just a patch. But npm@5 has package locks, so we can hash that and that's the exact code we'll be running. Maybe we can use those to override some deps and run forks of various modules if necessary? To get good results, it probably is necessary to be able to rerun old versions, because then we can make a fair comparison if we run the same benchmark on different hardware. (also - there are multiple factors which determine performance! memory, cpu, disk, network)

Hmm, so currently the modules being benchmarked are just dependencies of the benchmark... if we instead make the benchmarks into a module and pass in the deps then we could run new benchmarks against old code...

Yeah, I've been thinking about the different hardware aspect as well. If we share the output of the benchmark in a git repo, you would be able to see the performance on different machines, and easily see who the people were. There is also the node version aspect that we need to take into account.

Also this should be quite easy to turn into automated test reporting as well.

the current format for these benchmark scripts is basically samples of number of ops completed over time. so you have two columns - seconds and ops and just draw a graph. Can easily see if scalability is linear, polynomial, or logarithmic, and also how wobbly the line is (between JIT, os disk caching, memory allocation systems, etc it will be!)

Even just running hashes on various input sizes had all sorts of non-linear effects! http://dominictarr.github.io/crypto-bench/

If you just show a single number you don't see that!

Yeah interesting! That was what I was after with the multiple dimensions of graphs we talked about. The overall graph of a benchmark would be plotted against git commit and then you could zoom in a see the individual run. Maybe even overlaying a number of them to look for patterns. This combined with results from multiple machines/node versions could further provide data for figuring out what is going on.

Yes this will be super important! Without measurements, we don't know where we are going (applies to anything, but specially to performance).

@arj yeah, it turns out that the weird locking problem that @matt was suffering from recently was actually just a particular node version!

We've definitely observed some weird performance differences between UI versions also. Sometimes performance degrades slowly as you code (like a frog being gently boiled) and so you don't notice it subjectively.

:agree: (supposed to be green like Mikey's one)

I'll bite, I know this hasn't been granted, but as I said before. Profiling is a fun sport for me

I wanted to try and run a full profile of my sbot to see if there was anything sticking out. I've done some profiling of C++ code before and valgrind was quite nice there, so I fired up node --profile bin.js server. Let it shift through messages from tonight and got a profile. Got the information on profiling from here. One of the top offenders was:

4213 4.0% 4.1% LazyCompile: *module.exports /home/chrx/dev/scuttlebot/node_modules/epidemic-broadcast-trees/progress.js:45:27

Which is a bit odd for me. Don't mind the LazyCompile. The * means that the code has been optimized. But 4213 calls seems like a lot (it was running for max 5 minutes). I havn't looked at the epidemic broadcast tree code before. The progress.js is rather innocent, so its were its called that is the problem. index.js calls it in two places:

var onChange = opts.onChange || require('./bounce')(function () {
  console.log(progress(states))
}, 1000)

and

  progress: function () {
    return progress(states)
  },

So I added a console.log to the second one, as the first is decounced, to see and that function seems to be called around every 100ms or more after connecting to a peer. I guess that is a bit excessive?

How it this different with only me working on it for the January grant?

When I talked to dominic about the proposal, I proposed that we split it so that I would do the CI and visualization part and dominic would be working on more benchmarks and general optimizations. I wanted to make my part as minimal as possible as I wanted to have dominic working as much as possible on making ssb better. The way this changes things is that I'll expand the scope of the CI and visualization to make it more complete, but still have time to expand on the ssb-benchmarks already written. I also would like to work on %zaaPeF1... and %3+jTw8M... plus whatever else the benchmarks might uncover.

Hello @robert. So happy for you to join in!

My idea with the CI module was:

Write a CI module. I'm thinking it needs a few configuration knobs: repo, what to do, how often (sub-module changes, git commit, version bump). Then a first version can run on top of git-ssb and watch for commit and do whatever it needs to do. A first version will dump the result in a txt file that can be read from the visualizer and shared (probably in a git repo).

The first step would be to see if any existing CI frameworks can help us here. I'm for better a worse a not invented here kind of guy. Do you have any recommendations? As for running it I don't want to overcomplicate things, so probably something like a chroot. Docker might also be a good idea, I recently learning about the dynamic user feature in systemd 235, sadly is quite new so not widely supported. Its important for me that it is learn to keep the overhead low.

I have created an empty git-ssb here ssb://%ojQGCYUpXg0KPz8y1ou+m6XzCz/89O26QepBrv8vKjM=.sha256. It would be most helpful with an overview of existing solutions. But please don't get too attached to any of them Overall it seems to me that the CI part is a simple problem, and we need a simple distributed offline solution that works well with our existing infrastructure.

And yes would be great if the CI had some notion of testing as well, but for this specific proposal I'll try to limit myself to benchmarking, but maybe if you're interesting that could be something for you to dig into? There are already plenty of tests for a large number of the components.

As for GitHub, while a lof of the scuttlebutt parts is developed there, we do try to cultivate our own git-ssb. @cel has a grant for improving things and have added comments to commits, which is super awesome.

The bots aspect and automation is really cool. @dinosaur added a framework for writing bots. I plan on looking into that.

Yeah a lof of these CI solutions are very much focused on cloud first. I did find this that looks promising. Very few dependencies.

And thanks for the benchmark.js link, we need something like that for the statistical aspects. I have added it to the README.md I did yesterday in the benchmarking-ci repo, where I have tried to lay out an implementation plan.

Its interesting that we arrived at more or less the same conclusion about how we decentralize the running part. I don't think the solution is complex. You just share the results in a git-ssb repo.