What I totally skipped in that enumeration was the change to the combined indexing setup.
As a refresher, before indexes were loaded/mounted as plugins. There was one for
root:%yyy, one for
author:@xxx, one for
content.type:zzz. This worked mostly ok as long as they truly orthogonal to each other, meaning they don't influence or depend on each other. The only example that sticks out for which you can somewhat say this is
author, no amount other of information will change its mapping.
type: however look vastly different depending on the content of a private message. Additionally, the contact graph and about/names index would ideally define their filter criteria on
type: to read less messages but in the old system all the indexes read all the messages in the receive log individually, resulting in poor performance because of lots of overhead.
While I'm sure that it is possible to design an abstract stream processing system in which this can be cleanly modeled I took the opposite approach.
There is what I called a combined application index now which is in charge of filling
root while also supporting unboxing private message on all of those. This means it's finally possible to do
messagesByType contact private:true and put follow/block messages in groups or
tangles.replies %theThread private:true to read a private thread..! I'm quite excited about this since this always blocked go-ssb from being used as a real application bot, not just a replication server.
Since the combined index fills all the sub-indexes at once this also removed the huge read-overhead, resulting in about 1 minute to index what took at least 5 times as long before.