How can we deploy new transforms to Scuttlebutt?
The most prominent use-case is that we always want to decrypt messages before the view sees the message, so we transform the data before it's passed to the view. This is great! But what happens if we want to deploy a different transform to the data? Some examples:
- decryption with a different key
- decryption with a different algorithm
- decryption of a group chat we've just been invited to
- my use-case: off-chain content
If we add one of the above then future messages that we receive will be correctly transformed before they're passed to a view, but previous messages will still be in their decrypted (or unlinked) state. Consider the following:
- Scuttlebutt deploys group chats!
- Alice upgrades her client
- Alice starts an encrypted group chat with Bob
- Bob upgrades his client
- Bob checks his group chat view and doesn't see anything
The recommended way to handle this is to delete your views and rebuild them from scratch. This is slow, resource-intensive, and frustrating for everyone involved. I've talked to Dominic about partial rebuilds where we could rebuild an individual message but he didn't seem keen, and this still wouldn't solve the problem of reduce functions where you can't just rebuild an individual message.
We could deploy a small change to the protocol when we deploy the new transform function, which would stop old clients from replicating new messages so they wouldn't receive messages that their client couldn't understand. Once they update their client to the newest version, they'd receive (and transform) the new messages.
I'm not super excited about solving an implementation problem by soft-forking the protocol, but it's an option. This is a duct tape solution at best, but it might be better than forcing every client to do a full rebuild.
Maybe these transforms should be their own views? Maybe views should be able to query each other? Maybe partial rebuilds on map views are worth trying? Maybe we should be using SQL? Maybe there's an option I haven't considered? I'd really love to be able to deploy these improvements without breaking everything in the process.
I don't know
cc: @dominic @mix @andré @SoapDog @mikey @arj @regular @happy