Mutable Messages: Dev Diary

Applied to #ssbc-grants : %2HYP9MQ...
Accepted: %6jIFKhP...
Project repo: %NPNNvcn...

The work on this grant started with %mvd, and it continues today. I'll set the goal up now that I'd love to wrap this grant by August 20th, which is nearly two months from today. This means I'll have approximately three months to complete the grant goals, if I factor in the time I spent developing a mutable messages prototype.

I think this is a realistic goal, since I will continue to work at Levi's part-time while developing software the rest of the time. This is usually how I work, so I'm not changing anything about my life because of the grant. I'm only adding the benefit of being paid to work on the software that I care about, for the first time in my life! Awesome.

The stretch goal for this grant is ssb-wiki. Building a distributed wiki has been a goal since the early days of cypherspace, implementing a prototype would be an adventure.

This means that I have two goals with this grant. The first is to bring mutable messages to the ssb community, and the second is to implement an experimental ssb-wiki that will probably just be a dedicated client for publishing and editing wikis. If wikis also get implemented in other clients, that'll be beyond the reach of this grant.

What does 'bring mutable messages to the ssb community' mean? For me that goal means that I want to first finalize the schema for mutable messages, and then give the mutable message workflow a thorough spin around the block with a group of testers. Once I'm clear that I've implemented mutable messages right, I want to move towards implementing mutable messages in more ssb clients.

The easy A+ here will be bringing mutable messages to Patchless, because I'm intimately familiar with the code base. The challenging part will be bringing mutable messages to Patchcore, because I haven't developed with that code-base or used any of the Patchcore based clients since Patchbay@6. But Patchwork is the most popular ssb client, so this means to achieve the goals of this %ssbc-grant I must also implement mutable messages in Patchwork.

Ideally we could implement some sort of shared library that is accessible in all of these clients. But realistically I think all three have slightly different ways of handling message rendering, and thus might need three different opinions about how to render mutable messages.

Will I be able to implement mutable messages in clients beyond Decent, Patchless, and Patchcore? We'll see how much available time I have after getting mutable messages implemented in Patchcore.

Everyone: feel free to this this dev diary thread to talk about any/all things related to mutable messages. Or feel free to talk to me in public and/or private. Your messages will show up in my mvd queue!

Here's some scrots of the mutable message workflow on a mvd testnet:

Write a message:

Preview a message:

Published message:

Edit a message:

Preview your edit:

Published edit:

~

I shouldn't forget, this grant also promises to implement diff messages that show how a mutable message changes over time!

@ev

Nice! Some questions:

• Am I right if I say that this is mostly (from a technical pov) orthogonal to deletable messages/blobs-as-messages? Any design considerations you have to take for when/if that gets implemented?
• This should work fine with ooo, right?

@Powersource

This should work fine with ooo, right?

Yes, it should. But if you're using ssb-ooo to retrieve messages you'll also want to use ssb-ooo to check for edits to those messages. Right now I'm not aware of any clients that use ssb-ooo, are you working with one?

orthogonal to deletable messages/blobs-as-messages?

This work with mutable messages won't make it possible to delete messages. I'm still using the secure-scuttlebutt log to create messages and then edit them. Using blobs as messages is a good idea, and you could always use this mutable message implementation to edit the blob hash you're linking to in order to point to a new blob. But it'll always be possible to figure out which blobs you were linking to before, so that won't make them go away forever.

Let me know if that answers the question, if not feel free to ask more questions!

mutable messages update

Over the 4th of July weekend, I've been participating in a family event. My dad visited from Texas, so that was a bit of a time commitment over-and-above my job at Levi's.

One of the big accomplishments that I've made over the past two weeks has been rolling out the mvd lite client at http://gitmx.com/ -- it's been more than a year since I was able to get the lite client to work with the main network. This means that mvd is both lite enough to work over websockets, and also that scuttlebot is more memory efficient than it was a year ago.

http://gitmx.com/ is a great place to test mutable messages on the main ssb network, if you want to try it without cloning and installing mvd itself. I'd prefer if you'd clone and install mvd and try it on your local computer. But it isn't exactly a one-click-install the way that Patchwork is, yet.

To clone mvd using git-ssb try: git clone ssb://%NPNNvcnTMZUFZSWl/2Z4XX+YSdqsqOhyPacp+lgpQUw=.sha256

My schedule over the next few days is mostly clear. I'm hoping to spend some time implementing diff messages, so you can see how messages change over time.

ssb-wiki

Creating ssb-wiki was a stretch goal for my #ssb-grant. However, I decided to take a stab at making it a reality today by implementing a new post type, wiki. I also implemented the render, and associated buttons. So, it turns out ssb-wiki is ready to go.

You can see this work in my latest git-ssb commit

You can test out ssb-wiki now on a testnet at http://mutable.evbogue.com/ or clone down the latest version of %mvd: git clone ssb://%NPNNvcnTMZUFZSWl/2Z4XX+YSdqsqOhyPacp+lgpQUw=.sha256

Wiki posts are a little different from normal posts:

• Anyone can edit a wiki post (on normal posts, only you can edit)
• Edits are accepted from anyone within your foaf (friend of a friend) social network
• There is no way to 'reply' to a wiki, instead it can only be edited
• On the wiki thread you can see a list of all the associated edits to the wiki over time

Once diff messages are implement, those edit messages will look a lot sexier.

What's next?

• Implement diff message renders
• Clone down Patchwork and try to figure out how I'm going to get mutable messages and ssb-wiki merged into patchcore

@rabble yes, @kas has it right. leveldown isn't working with the latest version of Node. Use @creationix's nvm to install the LTS version of Node.js on your system. https://github.com/creationix/nvm

Once you have nvm installed, the command to install the LTS version of Node is nvm install --lts

P.S. Thanks everyone for trying mvd! I appreciate your effort to try new software.

Please continue to report bugs and/or obstacles here, no matter how mundane they might seem. There are still plenty of kinks to work out.

I'm making progress implementing diff messages in %mvd

I made some progress showing the history of edits in %mvd. Here's an ssb-wiki example published to a testnet:

Also, edits now link back to the original post.

Great progress! Really enjoying this thread @ev.

@ev

I worked on some of the Node 10 compat stuff for Patchwork/Patchbay, but this looks way eaiser -- it looks like the only dependencies that require the old leveldown are unused, so I was able to mess a bit with the dependencies and everything is looking great under Node 10. I'll push up PR in a few minutes.

This is my first time using mvd and it's working great!

@christianbundy Awesome! I'm taking a look at your pull-request. query/index.js is a fork of ssb-query until https://github.com/ssbc/secure-scuttlebutt/pull/215 is merged.

@christianbundy Merged in %HF+1Pfg... ! Thanks for getting mvd working with node@10

This should solve your issue @rabble

Does scuttlebot@latest run on node@10 now?

@kas Yep, works great for me and Travis!

Oh great, then I can skip nodejs-lts-carbon and just use Arch's rolling release. Right?

@kas.v2

You got it. That's what I'm doing and I haven't had any problems at all. The only "problem" I've seen is a few deprecation warnings when running npm test, but they're just warnings and I've already opened a pull request to resolve them.

@bundy,

Thanks. It seems to run fine, but I am now seeing (perhaps I also did before?) ssb-friends: stream legacy api used. I only use EBT gossiping and I have installed @cel's ssb-ebt-only plugin.

@kas I'm seeing that as well, but I have to admit I'm not famliar enough to know what it means.

@bundy,

Thanks.

I have one problem: The ssb-fulltext package, somewhere within its Russian dolls hierarchy, requires or installs leveldown@1.9 which cannot be built with node@10. I can run patchfoo with ssb-fulltext disabled but it's suboptimal. I believe it happens somewhere in the levi submodule, but there's a lot of leveldown going on there:

$ find ssb-fulltext -type f -name package.json -print0 \
  | xargs -r0 grep leveldown > ssb-fulltext_fscked_leveldown.txt

ssb-fulltext_fscked_leveldown.txt

Any suggestions?

@ev How difficult would it be to have mutable messages in other clients (Patchwork, Patchbay, etc.)? I'd love to be able to use these, but I'm not sure whether it's possible or practical.

@ev this is awesome! really cuts down the scope of multiwiki. kind of great because i've been rabbit-holing a bit getting my scheme environment ironed out. at the very least, it will be great to have a native ssb client.

@Christian Bundy It's both practical and possible.

I'm waiting for the dust to settle in @matt's Patchwork debloat before diving into the code. Patchbay is unclear from a political standpoint, for obvious reasons -- maybe you can help here?

Mutable messages are an additional query against each message to see if there are any edits to the message. Edits are a new message type type: edit. There's no reason why we can't get mutable messages working everywhere, now that they're prototyped mvd.

@ev

Maybe a silly question, but could the edit code be used as a Scuttlebot plugin? For example, the addMap() function seems like it would make edit messages readable to all clients.

I'm also super curious about the performance, it seems like doing two lookups for each message might be pretty resource-intensive, but at our scale that might be negligible.

@Christian Bundy Maybe? I'm up for anything that would make mutable messages faster.

Doing a query against a message is slower, so if there's a better way to find message edits I'm down for merging it. I'd love to speed up every ssb-query actually, because when you start querying things get slower.

What does the addMap() function do? This is the first I've heard of it.

@ev

The background on addMap() was that we needed I wanted a way to prototype blob content, so a map feature was added to flumedb so you could modify messages on read. For example, you could implement cloud to butt with something like:

var db = Flume(log, true, (val, cb) => {
  if (val.value.content.type === post) {
    val.value.content.text = val.value.content.text.replace(/cloud/g, 'butt')
  }

  cb(null, val)
})

This function runs on every db.get() or db.stream(), which gives you the ability to modify messages before they're accessed by other plugins. You could also do something like "if val.value.content.type !== 'edit', check for edits". Unfortunately flumedb only allows a single map function, which in our case is passed by secure-scuttlebutt, so the PR I linked is meant to expose a way for plugins to add their own map functions. There's an example on the branch, and if my client was running that code your above message would appear on my client (and to all of the plugins) as:

{
  "fizz": true,
  "key": "%STbDG1a7WAseq/E40XuPxeMjEqWOhSamjUfjtt5Hfr8=.sha256",
  "value": {
    "timestamp": 1539198459294,
    // ...
  }
}

TL;DR: it's a way to call a function and modify messages on read without having to interface with other plugins.

See also: ssb-friends: How to get non-legacy methods?

@ev, thanks for the links. It would be so sweet to be able to write wikis – I wish more clients had support for edits.

@ev @kas

I've extracted the edit logic and turned it into a small ssb-server plugin. It has the same caveats as ssb-blob-content (requires { private: true }, doesn't play nicely with views), but it seems to work well enough as a prototype.

• GitHub: https://github.com/fraction/ssb-edit
• git-ssb: %qDZyY87...

cc: #ssb-server