You are reading content from Scuttlebutt
@Anders %OaT6TgLAAg1098bGtZqr6tDn9mNUm2/3Ke2idmYtIDc=.sha256
Re: %HqvC29wmr

Sure.

There are many areas we have improved the performance. Starting from the lowest level the flume log we have these numbers:

async flumelog:

name, ops/second, mb/second, ops, total-mb, seconds
append, 923964.807, 138.002, 4620748, 690.149, 5.001
stream, 1059075.865, 158.182, 4620748, 690.149, 4.363
stream no cache, 1102803.818, 164.713, 4620748, 690.149, 4.19
stream10, 2540947.641, 379.51, 12714902, 1899.068, 5.004
random, 39715.656, 5.931, 198618, 29.664, 5.001

flumelog offset:

name, ops/second, mb/second, ops, total-mb, seconds
append, 306180.037, 45.74, 3064556, 457.817, 10.009
stream, 294511.348, 43.997, 2945408, 440.017, 10.001
stream no cache, 327724.949, 48.959, 3064556, 457.817, 9.351
stream10, 452973.302, 67.67, 4530186, 676.776, 10.001
random, 28774.712, 4.298, 287891, 43.008, 10.005

Next there is how we handle encrypted messages, where the overhead is constant more or less, meaning it doesn't matter how many indexes you have (https://github.com/ssb-ngi-pointer/ssb-db2/pull/46) . And for private groups the improvements should be even bigger.

Then there is the way the indexes are handled. Currently we have more specialized indexes, I don't have raw numbers directly for those. This will get even better once we incorporate the prefix indexes. Overall these index improvements also means smaller indexes so we save some hd space.

Overall I think we could end up with something like 10x improvement once things are settled. We should know more in around 3 weeks. I plan on using the excellent fixtures to see exactly how much we get. Currently head deep in the implementation of all of this.

Another thing that I'm quite excited about is how the go implementation is coming along. I did some benchmarks a few months ago and the improvements for replication are quite good. I'm also looking forward to changing my server over to the go bot once we have EBT. @cryptix is working on that now.

Remote go:

full feeds: 2.797s
partial feeds: 30.752s

Remote js:

full feeds: 4.026s
partial feeds: 51.746s
Join Scuttlebutt now