this sounds like a cool area to work on @craig. It's isolated, and of a size that you can nail somehting really well that we can all use.

I've got a branch on patchbay called mutant-scroll. It uses a module called mutant-scroll which I wrote for Ticktack, and want to use here. It's an attempt at a seperation of rolled up data and views. The major questions about it is what happens when that data structure gets BIG. Anyway I'm pretty happy with the work and this is behaves pretty intuitively in patchbay. Would love to talk about it sometime if it's of interest / use

am I correct in assuming that the current rollup code in patchwork or patchbay does not use Flume to create a record of the rolled up messages?

@craig i believe yes, you're correct!

@matt described his last re-write of the rollup algorithim here:

Here's a quick overview of the new algorithm:

• In batches of 2000 posts taken from the user asserted timestamps (sbot.feed order), group posts by thread.
• When a thread does not have a root message (in this batch) and it is the public feed, remove all replies from strangers
• Bump threads upwards by most recently received message (sync time, sbot.log order).

It also makes improvements to grouping of "follows" and "friends" trying to reduce the number of feed events needed to communicate the contact changes.

naively, a flume view makes sense.

keen to hear what @matt thinks, since he has the most experience in this domain (so many edge cases!), and i know has some ideas for how he wants to re-approach our infinite streams.

@craig oh wait, there's another more recent rollup re-write using flume views here and here.

looking at patchcore/modules/feed/pull/rollup, which is called by patchwork/modules/feed/html/rollup, which is called by patchwork/modules/page/html/render/public, it looks like the primary flume view is patchwork/sbot/roots, then secondary flume views are used to attach corresponding data.

wee!

love that name.

considerations for #rollway :

• offer different ways to sort by
  • unread
  • most recently updated (with a post, not a like)
  • time of original post
• make ways of filtering:
  • unread
  • just see threads my friends (or a set of named people)
• how are updates handled?
  • when does it happen
  • what triggers it
  • do you have duplicates of posts in the feed
• which direction is "new"

here's what the flow of data looks like for mutant-scroll

e.g.

1. the topStream is triggered when scrolling up (in my case this is the 'live' stream of recent data)
2. this hands new data to the topReduce, which mutates the record of the state
   • here it checks inf there's an existing entry for the card, and if there is, moves that entry to the top of the state array.
   • if there's not, then it just prepends the new card
3. the state is being perpetually mapped into the view (by mutant)

You can see this code in : https://github.com/ssbc/patchbay/blob/mutant-scroll/app/page/feed.js

If the code looks messy - it is, this is a total MVP hack to see how it feels to use.

oh I called them updateTop not topReduce because it didn't follow a classic reducer signature because mutant...

Btw, @craig since you'll obviously appreciate it, I spent a lot of time researching forrestry terms before I came to the name flumedb - so totally approve of rollway.

thoughts (from the most databasey perspective):

1. Index that maps msg_id -> messages linking to that message (stored as a hashtable -> an array of sequence numbers is most compact, each sequence number is only 4 bytes)

2. index that links most recently linked message, this would essentially be a doubly linked list where you move things to the front. (doubly linked so you can take an item and move it to the front, then relink the items it was previously between)

These two indexes combined allows you to scroll back through patchwork style indexes by looking at the second one, then the first one. Since each index is not specifically about threads, it works for every message type, you just filter for posts of the particular type you need.

Just chiming in to say I am so delighted by this process @Craig, thanks for trying out this approach with us.

Also I think a cool name is pretty important.