Replacement

As discussed above, replacing the item with a fake message should work, although it may be creating errors. The biggest problem right now is that not all flumeviews support rebuild(), so we don't have a way to force rebuilds without restarting ssb-server. I've pasted some of the view errors I'm seeing, so maybe we should be outputing warnings when a flumeview only has partial API support?

Merge delete

Assuming we have JSON items A, B, and C, one way to delete B would be to add a bunch of whitespace at the end of A so that it overwrites B. I was having problems where my writes weren't being respected, maybe due to the multiple layers of caching, but this seems possible.

Unfortunately it's hacky as hell and only works if you can pad the previous item with whitespace, plus you have to do the opposite with the first item since there isn't a previous item to merge into. Another downside is that if you request B then it seems to just output a corrupted string -- again, I imagine this is a cache thing but I wasn't able to circumvent it while working on the problem yesterday.

Skip delete

When we get/stream an item we're using a getMeta() function that works with pull-cursor to iterate through the file. This function returns the value of the requested item plus the sequence numbers of the items before and after the current item. One hacky and less-performant idea was to read the before/after values, and skip them if they're null.

For example, assuming A - F are arbitrary messages, we delete C and then request D. The function should return the current value of D, notice that C is an empty buffer, and return B and E as the before/after sequences. This is fine, but doesn't work at the beginning or end of the file. It also means that we need to read at least 3x as much data on each read, plus buffer comparisons, which I'd imagine is prohibitively slow (unless we add another layer of caching).

Misc

Again, I'm super impressed by the speed of flumelog-offset, and it's way cool that it can write at 13.8 MB /s, but I've been watching the file size and I've never seen it increase faster than 800 KiB/s because the flumelog isn't the bottleneck -- the network and indexes are.

The offset db is incredibly fast and efficient, but it feels like we've traded away mutability for some speed gains that we don't actually use. I'm sure it's possible to add mutability to flumedb, but when working on it I feel like I'm really cutting across the grain. This might just be another incantation of boring Scuttlebutt, but since we're nowhere close to the theoretical speed of flumelog-offset then I'm wondering whether it'd make sense to use something more... boring.

cc: @cel