You are reading content from Scuttlebutt
@cryptix %lcjr+sF+ZLtEwnIfIObuf99nG5ps86qPzW8VbA3yP/Y=.sha256
Re: %i6pDs9WVW

What I totally skipped in that enumeration was the change to the combined indexing setup.

As a refresher, before indexes were loaded/mounted as plugins. There was one for root:%yyy, one for author:@xxx, one for content.type:zzz. This worked mostly ok as long as they truly orthogonal to each other, meaning they don't influence or depend on each other. The only example that sticks out for which you can somewhat say this is author, no amount other of information will change its mapping. root: and type: however look vastly different depending on the content of a private message. Additionally, the contact graph and about/names index would ideally define their filter criteria on type: to read less messages but in the old system all the indexes read all the messages in the receive log individually, resulting in poor performance because of lots of overhead.

While I'm sure that it is possible to design an abstract stream processing system in which this can be cleanly modeled I took the opposite approach.

There is what I called a combined application index now which is in charge of filling author, type and root while also supporting unboxing private message on all of those. This means it's finally possible to do messagesByType contact private:true and put follow/block messages in groups or tangles.replies %theThread private:true to read a private thread..! I'm quite excited about this since this always blocked go-ssb from being used as a real application bot, not just a replication server.

Since the combined index fills all the sub-indexes at once this also removed the huge read-overhead, resulting in about 1 minute to index what took at least 5 times as long before.

Join Scuttlebutt now