You are reading content from Scuttlebutt
@andrestaltz %Y6LwwMA7iYZ8KIlJzN23lXREw/So07ZcJAxZazSNHUA=.sha256
Re: %HqvC29wmr

Yesterday I worked 10 hours (hold onto your seats, precious hands, don't go RSI-ing again!) on various things ssb-db2, bug fixes, etc. Surprisingly, the bugs just keep coming one after the other.

I also wanted to quickly experiment with leveldb alternatives. I looked into cacache, which is a persisted cache built by the npm team, supposedly "really fast, really good at concurrency, and it will never give you corrupted data". I tried using cacache for storing the mentions of each feed. So key=feedid, value=array-of-msg-sequences. Nope, it wasn't nearly as fast enough as leveldb, in fact the query didn't complete within 60sec. So I dropped that.

Then I gave level-rocksdb a shot (RocksDB is a fork of leveldb built by Facebook, meant for servers). It was basically 10% slower than normal leveldb. Then I tried level-mem which has no persistence, it does everything in memory, I wanted to try that just to see if working with things in memory makes anything faster. Nope. So I decided I'm done with leveldb experiments. And cacache too by the way.

I began thinking about inverted indexes again, but it was a lot of work to experiment with that. I kept thinking about the problem of mentions. With cacache, it behaved a bit like your blobs folder: a bunch of nested folders for various hex prefixes, e.g. ./a8/77/theactualfile. And with that I think it made sense to me that each of the files should be small, so it's quick to save as well as quick to load into memory, and doesn't occupy a lot of memory when loaded. And that reminded me of jitdb bitvector indexes, which are approximately 150KB each.

So that brought me back to basic jitdb bitvectors, and I realized that the primary difficulty with mentions is that they're an array, while root or are just plain strings, which allow you to do a simple EQUAL comparison. So I built an INCLUDES operation in jitdb that checks if an array contains stuff. To build that, it required me to fix some tiny issues with bipf. It worked! Then, because mentions are usually not array of strings, but instead are array of {link,name} objects, I made an additional pluck function that allows you to compare a field inside each array item. And it worked too! I hacked that into ssb-db2 almost near midnight, and ripped out leveldb. I'm now thinking what else can be done with jitdb...

Join Scuttlebutt now