flumelog-aligned-offset

I started this a while back, but recently got it working properly! It's a new flumelog format, that is a little different from flumelog-offset. Like flumelog-offset there data is read as blocks, but what is different, is that records never overlap blocks. At the end of each block, there is an empty space, but it means you can always read any record from a single block, and don't need to copy memory when reading an overlapping block. Also, it's implement on top of dat's random-access-storage which means it works in the browser!

It's also much faster than flumelog-offset.

offset:

name, ops/second, mb/second, ops, total-mb, seconds
append, 142319.536, 18.007, 711740, 90.055, 5.001
stream, 288621.248, 36.518, 711740, 90.055, 2.466
stream no cache, 361106.037, 45.69, 711740, 90.055, 1.971
stream10, 312574.815, 39.549, 1568813, 198.498, 5.019
random, 63731.96, 8.063, 318851, 40.343, 5.003

aligned-offset:

name, ops/second, mb/second, ops, total-mb, seconds
append, 471424.83, 59.623, 2358067, 298.236, 5.002
stream, 1137503.617, 143.867, 2358045, 298.236, 2.073
stream no cache, 915392.08, 115.772, 2358050, 298.228, 2.576
stream10, 1201254.247, 151.928, 6009875, 760.098, 5.003
random, 355715.056, 5078.074, 1778931, 25395.448, 5.001

append, stream, stream no cache, stream10 are all 3x faster. and random is over 500 times faster (not sure why, yet)

unlocks binary format achievement

Awesome! I don't understand what block means but the fact that it's faster is very exciting. Any approximate plan how would we migrate ssb-db from flumelog-offset to flumelog-aligned-offset?

@andrestaltz your file system is built on top of a hard drive, also known as a "block device". A block is a kinda like a city block. it's a fixed sized unit, with many addresses inside of it. when the operating system wants to read some bytes from a file, the hard drive returns the entire block, most of which the OS discards. So, by designing a thing that aligns (I accept responsibility for that pun) with that means you save quite some extra work. Especially since memory also works like this (except memory blocks are called "pages")

I think the main perf difference in this case is avoiding several observables that were getting called every time you read a value from flumelog-offset.

... okay it benchmarks fast but linking it into ssb-db I'm getting some test failures... back to work.

@dominic

This is super rad, nice work! I remember reading somewhere that someone had success with keeping the data in a separate file from the metadata -- is that something that you experimented with?

Okay I got it working now. All tests are working with flumelog-aligned-offset in place. I had a really bad race that I hadn't caught in the tests so far. Also, I discovered that ssb-ebt tests don't pass if you are not on the wifi. Something about the peers don't get addresses and so they don't connect to each other.

The api isn't 100% compatible with flumelog-offset? I tried replacing offsetlog in ssb-db/minimal.js to test and I hit:

/home/chrx/dev/bench-ssb/node_modules/flumedb/wrap.js:31
      since.once(function (upto) {
            ^

TypeError: since.once is not a function

When I try running 04-minimal in bench-ssb.

Nevermind, found the compat file. Here are some numbers from bench-ssb for one of my faster machines:

hmm, improvement is noticable on flume test, but it's overwhelmed by JSON and all the other stuff slowing it down...

Yep, its probably validate. This will be hard to do anything about, except cheat a bit.