@dominic

hmm, a way simpler way than streaming deletes to the views, would be to filter out deletes from the views when querying them. views such as flumeview-level call to the log to get the message - if it's not there, drop it, done.

That's a really interesting idea, I hadn't thought of that. I think that makes a lot of sense, but I'm having trouble trying to grok how to:

• know when all items deleted from the log have been deleted from the view(s)
  • because since won't change, I'm not sure how we could be sure items are deleted
  • because items are only deleted on get(), could a rarely-touched item survive indefinitely?
• rebuild all views for a thread when we receive information to correctly build a view
  • because of the sub-list above, could we be sure the view for an item is correctly rebuilt?
  • for example:
    • unbox keys (encrypted groups)
    • message edits (mutable messages)
    • blobs (off-chain content)

Also, the bit about since in logs makes so much sense. Do flumelogs follow the same pattern? For example, do they always increase as an integer or do they follow the format of the log (e.g. a timestamp)? I was originally thinking that rebuilds could be triggered by del() and an immediate re-append(), but I just noticed this in the flumedb readme:

a flumeview must process items from the main log in order, otherwise inconsistencies will occur

Are these inconsistencies dangerous, or are they things like item.seq + item.value varying because there's no guarantee that you'll always have the same seq and value matched up?

@regular

That's so rad, thanks for pointing that out! I love that you're forging ahead and you've already worked on reversible reduce functions. Your message didn't come in until I'd already started this message, but here's what I originally wrote in reply to what you quoted:

True, it's possible to write the inverse of some [trivial] reduce functions, but I'm happy to take some easy performance wins before even trying to think about that.

P.S. You may already know, but on the latest Patchbay/Patchwork you can start your fenced code block with ```js and it should do syntax highlighting. It should also work with most other languages.