Here is a strawman proposal for what it could look like if we go (moderately) crazy on partial feed replication rpcs:
In current ssb, a replication request consists of a feed id and the newest sequence number on that feed known to the requesting server. If both peers have that feed, but their newest known seqnum differs, the peer who knows about more messages sends the new ones to the other.
A partial replication request would additionally contain the following values:
- range: exactly one of
- all: don't filter by seqnum
- start(amount): only exchange messages with sequence number below
amount
- end(amount): only exchange message among the newest
amount
many- note that this results in different ranges for the two peers, but that's by design
- types_exact: A set of utf-8 strings. Only exchange messages whose type is in the set, unless the set is empty.
- the set can be stored as a radix tree (or really any compression scheme)
- types_prefixes: A set of utf-8 strings. Only exchange messages whose type begins with a string in the set, unless the set is empty
- the set can be stored as a radix tree (or really any compression scheme)
These values act as filters, only messages that pass all filters are replicated.
This scheme allows encoding a reasonable subset of all possible feed partitions. For fully general feed partions, you can still expose replication rpcs via plugins that use merkle-ooo to do the actual message exchange.
@andrestaltz This scheme would be sufficiently expressive for all your manyverse needs, right?
And a strawman proposal for allowing applications to use message schemes where state is reduced over multiple messages of the same feed:
Each message metadata contains up to eight integers, each smaller then the sequence number of the message. When replicating a message (e.g. via partial replication, merkle-ooo, or transitively), the messages from the same feed whose sequence number is one of those integers are also replicated.
Huh, that turned out surprisingly simple.