Scuttlebutt Guide

Looking to learn how to build in Scuttlebutt? There's currently no canonical resource, but here's a map of the known archipelago!

Have you found an Uncharted Isle? We'd love to hear about it.

I'm starting to aggregate resources for learning how to biuld on Scuttlebutt. There are currently lots of resources out there, but they're spread (rightly) like an archipelago. I intend to aggregate as many as possible, and perhaps this can inform some awesome resource in the future.

Asks

• flick me links or PRs to sweet resources !
• ask for things you want (I'll get it made then add it) !

github : https://github.com/ssbc/scuttlebutt-guide
git-ssb : ssb://%rRmgg8grYJ/upfXKxFNG62Y49CmkEShk70HoFLIEyDY=.sha256

@noffle. I made a flume index for chess games here: https://github.com/Happy0/ssb-chess-db (it's not the most efficient data structure, but it'll do for now since there are relatively little chess games :P.) I thought I'd quickly write down what I know from having done it. Maybe I could write a tutorial where I take the time to make sure it's understandable, but for now we can just have a conversation about it :P.

Note: my index is 'in memory' using flumeview-reduce ( https://github.com/flumedb/flumeview-reduce ). I'm not sure how to keep it on disk (it's something I was also confused about when I started building an index for chess games and I also came across the current docs thing you linked to and didn't understand it, so I decided to just do it in memory using flumeview-reduce for now.)

Perhaps flumeview-reduce will work for your use case, so I'll write what I know about it (including scuttlebot plugins which you probably already know a bit about):

A scuttlebutt plugin should expose an 'init' function that accepts 'ssb' and 'config' params. ssb is the scuttlebot instance, and config is its config. The init function should return an object of functions that can be invoked on the plugin via the scuttlebot instance (via mux-rpc).

The plugin should expose a manifest via a exports.manifest file that describes the mux-rpc (https://github.com/ssbc/muxrpc) functions that your plugin exposes and their return types. See the mux-rpc github page for more details of what those return types can be.

It should also expose a plugin name and a version. There are some gotchas about naming your plugin that I don't really understand. More details about my confusion in this thread: %fYE4sen... . Since @arj found a workaround and documented it on patchaby's github page, I moved on from investigating it. Maybe I could return to it (especially if I do try to write a tutorial.)

Those functions can then read the view created by the index and process it in useful ways.

If you're building an in-memory index, you should call sbot._flumeUse, give it the name of the index as the first parameter, and as the second param a flumeview-reduce object with a mapper and a reduce (i guess there may be other things you can pass to flumeUse that would use the disk). The mapper and reducer you supply are called for each message in your scuttlebot database (and newly arriving ones), and your reducer should build up your index (in my case, a dictionary of chess games with a game ID as a key and the game status (invited, playing, ended)).. _flumeUse returns a 'viewobject, which you can callview.get` on to get the current state of your index as a callback.

The index is written to the disk periodically, so next time you start scuttlebot it will start processing messages from the last one it has processed. If you change the layout of your index, you can bump the version param given to flumeview-reduce and it will process all the messages in your database from scratch again.

hey @vtduncan I'd love to add links our to the resources you're making. Do you have decentraliseable versions of your docs people can clone with git or similar?

hey @vtduncan I've pushed your work into a repo in the scuttle-cloud. You can push there too !
See this PR : https://github.com/vtduncan/scuttlebutt-protocol-guide/pull/1

nice question @paidforby_

Something not clearly written is that you can provide an ENV variable which starts you a different identity

app_name=mix_test_2 sbot server

this creates a new folder ~/.mix_test_2 in this case. Problem is this doesn't start the sbot on different ports, so it will collides with another running sbot.

@don did some work in a similar direction as you using docker : https://github.com/don-smith/ssb-tutorial which you might find interesting.
Nice accronym use it !

No problem @noffle :).

I think it must be flumeview-reduce itself doing the periodic write to disk. I don't know how periodic it is though - whether it's on every message, etc. I'm doing guess work based on the following:

The index is in memory (I know this because dominic told me it is :P) - but I know it must also be written to disk because when you load the app back up it only starts updating the index again from the last unprocessed message, and applies it to the current state of the index. It must load the index at its current state into memory again from disk, then apply the reduce function to it on newly arriving messages.

If it didn't write the state of the index to disk, and the last processed message, it'd have to process the feed stream from the start again.

I assume that accessing the index via view.get must be callback based (even though it's in memory) because you could pass something else (not flume view reduce) to ._flumeUse that's an on disk index (or that it will only be called back after the current reduce operation has finished.)

I was reading the code for flumeview-reduce yesterday - there were comment about how it wait for a minute of no new messages then writes (or similar)
https://github.com/flumedb/flumeview-reduce/blob/master/inject.js#L57-L63

flume uses seq (sequence) and upto to track wherre it's up to in terms or reducing (or materialising) a view. In scuttlebutt this might be a the timetamp that's being used. seq is (also) the keyword used for the location of the data in the store - in the case of the flumelog-offset it's a location based on bytes ! so your seq progresses like : 9, 27, 102, 344, 392, ... (based on how big the things were you logged)

hmmm .. I can't actually see the code for "waiting a minute" could be a dead comment.

There is this though : https://github.com/flumedb/flumeview-reduce/blob/master/inject.js#L145-L147
"If reduce is up to speed with log, then write."