@Daan @Daan: as part of the #sunrise-choir, @Piet has been implementing #flumedb in Rust, as i think it is a good abstraction for Scuttlebutt, especially if we combine it with sqlite to replace our existing custom query system.

for context, flume has two concepts, flume "logs" and flume "views". a flume "log" is an append-only store where you can only add messages and get them back by "id" (which depends on the log). in Scuttlebutt, we use a a log.offset file as our log, where messages are appended to this file and the id of a message is the offset from the beginning of the file. a flume "view" is similar to a materialized view in other databases, where the view is able to listen to each new message (once and only once, as guaranteed by flume), use each message to populate a store of indexed data, and then use the indexed data to answer questions on-demand. currently, Scuttlebutt is mostly based on ssb-backlinks and ssb-query, which use leveldb under the hood.

last week @Piet and i met with @Matt and @Mix to understand the minimum functionality that the #ssb-clients need in order to use our rusty flume. after that call, since Piet had already implemented the a memory log and an offset log, we decided to prioritize implementing a flume view using sqlite, which could replace ssb-backlinks and ssb-query and all the other bits.

so yeah, i agree we should keep our abstractions as conservative as possible, but i think flume is the correct abstraction, as conceptually Scuttlebutt is an append-only log of messages that you can ask questions to. i think using only sqlite would be a negative regression, but using sqlite to replace our custom query system is rad.