There are many areas we have improved the performance. Starting from the lowest level the flume log we have these numbers:
async flumelog: name, ops/second, mb/second, ops, total-mb, seconds append, 923964.807, 138.002, 4620748, 690.149, 5.001 stream, 1059075.865, 158.182, 4620748, 690.149, 4.363 stream no cache, 1102803.818, 164.713, 4620748, 690.149, 4.19 stream10, 2540947.641, 379.51, 12714902, 1899.068, 5.004 random, 39715.656, 5.931, 198618, 29.664, 5.001 flumelog offset: name, ops/second, mb/second, ops, total-mb, seconds append, 306180.037, 45.74, 3064556, 457.817, 10.009 stream, 294511.348, 43.997, 2945408, 440.017, 10.001 stream no cache, 327724.949, 48.959, 3064556, 457.817, 9.351 stream10, 452973.302, 67.67, 4530186, 676.776, 10.001 random, 28774.712, 4.298, 287891, 43.008, 10.005
Next there is how we handle encrypted messages, where the overhead is constant more or less, meaning it doesn't matter how many indexes you have (https://github.com/ssb-ngi-pointer/ssb-db2/pull/46) . And for private groups the improvements should be even bigger.
Then there is the way the indexes are handled. Currently we have more specialized indexes, I don't have raw numbers directly for those. This will get even better once we incorporate the prefix indexes. Overall these index improvements also means smaller indexes so we save some hd space.
Overall I think we could end up with something like 10x improvement once things are settled. We should know more in around 3 weeks. I plan on using the excellent fixtures to see exactly how much we get. Currently head deep in the implementation of all of this.
Another thing that I'm quite excited about is how the go implementation is coming along. I did some benchmarks a few months ago and the improvements for replication are quite good. I'm also looking forward to changing my server over to the go bot once we have EBT. @cryptix is working on that now.
Remote go: full feeds: 2.797s partial feeds: 30.752s Remote js: full feeds: 4.026s partial feeds: 51.746s