You are reading content from Scuttlebutt
@mix.exe %qPgEBZHnARoD5d+X8x2zxOvKg3XJyNAn45ZVKTHI/XM=.sha256

Mix's dev diary 2021


It's been a while since I posted about what I've been building in any focused way. I've been inspired by the NGI teams posts to revisit this practice. I also think it's important because many of you likely don't see the scuttlebutt work that's being done with #ahau so clearly, and there's some great things coming out of it.

I might post a summary of the rest of 2021 at some point, but to get underway I'm gonna start posting from the present

#dev-diaries #dev-diary
:art: art by Raphael Lacoste

@mix.exe %RmwONgaktnMIe8SSvtU5n97aC1a+PeegvadSzz+WOpA=.sha256

:key: ssb-keyring

A JS library extracted from ssb-tribes which is useful for persisting and synchronously querying for encryption keys.

Extracting it was following a pattern @arj started by extracting ssb-private-group-keys/. It feel really good to give this module some space .... I've expanded it into clearer domains and broken the tests out to match those domains.

It was also motivated by the drive to add "P.O. Box" keys to this store without it becoming a total mess. This is part of a bigger epic for #ahau - Subgroups. Our first feature that we're driving towards is kaitiaki-only subgroups, with the added functionality:

  • people can send messages to the kaitiaki (without being part of their group)
  • people can share maintenance of personal details (e.g. home address) with kaitiaki
    • this is a feature I'm pretty excited about. The intent is this offers privacy from the general group AND transparency so data you've shared is still "yours"


  • kaitiaki - guardian
User has not chosen to be hosted publicly
@Rabble %pR6YgxjLbDwSv07IAp2t+TacvXpBu8cXF3f5QeLnsHc=.sha256
Voted # Mix's dev diary 2021 ![raphael-lacoste-mushies-final-net.jpg](&X9VEtXix8
@mix.exe %6f3eDHvxEdVz9SRrDA2zpk0Frak3nEh1+vEm5cBrA1I=.sha256

ssb-crut v3.1.0

Adds a new feature which allows you to attach a record tangle to an arbitrary root, for example attaching a settings record to a gathering, or group

// specify the behaviour of our "settings" record
const spec = {
  type: 'settings',
  arbitraryRoot: true,  // << NEW!
  props: {
    admins: require('@tangle/simple-set')()
const settings = new CRUT(ssb, spec)
// update the settings record attached to a gathering
const updateId = await settings.update(gatheringId, {
  admins: { add: '@mixoxcviuoiu...ed25519' }

// or a group I'm a member of!
const updateId2 = await settings.updateGroup(groupId, {
  admins: { add: '@mixoxcviuoiu...ed25519' }

CRUT supports promise + callback style thanks to @Luandro Pàtwy

We're looking to use this in #ahau because it an elegant way to e.g. register the poBox of a group and attach it in really easy to look up way.

@mix.exe %b9dizvzCvE5CBwpgEz0E3B2WSZ50LCHigHTSioqdWs4=.sha256

ssb-crut v3.2.0

Adds support for the new ssb-db2 and it's indexes (instead of ssb-db and ssb-backlinks).
Thanks to @arj for this new functionality!

@mix.exe %xM4EyKBlJ/jQJu9kmMXQLx/iz3koVgoDI31oCQjtHHo=.sha256

ssb-tribes v2.3.0 - subtribes!

This adds the ability to mint private groups which are "inside" another group:

in practice just another group EXCEPT a link/group-subgroup message is published in the parent group pointing to the existense of a subgroup

We've also added the ability to mint groups with a P.O. Box, or to add one later

  • ssb.tribes.create({ addPOBox: true }, cb)
  • ssb.tribes.addPOBox(groupId, cb)

both of these publish a group/po-box message to the group, which transmits the P.O. Box keys to other members of the group so that they can use open private messages sent to the group from people outside the group

Example usecase

In ahau we are starting all groups with a "kaitiaki subgroup" - this group will be responsible for approving applications to join the group, and allows application details (and any discourse) to be private to the kaitiaki and applicant.

After an applicant is approved, they will be added to the main group, but the approving kaitiaki will also mint a "kaitiaki only" version of the profile (which the person who it's about will also have read/ write access to), allowing safe storage of contact details in a place not public to the whole group.

User has chosen not to be hosted publicly
@mix.exe %AgwxaeZiEgZIlDHXTb411uJlANQ/pLAeg0KwlcUETaw=.sha256
Voted yall are crushing it @mix.exe!!
@mix.exe %dj//oETd6DrNu41w1GMzvOobY4SrUAFu4/er6H9rb2c=.sha256

@colin (an academic helping #ahau out) is working on merging tangle branches, which is a surprisingly tricky problem! We noticed that writing tests was a grind, so I copied the Mermaid format for flowcharts, and made a little tool!


To build a graph of nodes like:

  / \
 B   C
  \ /

assuming direction is top->bottom

const { buildGraph } = require('@tangle/test')

const nodes = buildGraph(`

// => {
//   A: {
//     key: 'A',
//     previous: null
//   },
//   B: {
//     key: 'B',
//     value: 'cat',
//     previous: ['A']
//   },
//   C: {
//     key: 'C',
//     previous: ['A']
//   },
//   D: {
//     key: 'D',
//     previous: ['B', 'C']
//   }
// }
@mix.exe %TQUO2zaj+CLlag6znAnIJfP/EPynoZif7LIQ1kVRoBk=.sha256

More context: the @tangle/* repos are generic tools for reducing Directs Acyclic Graphs (DAGs) which contain Operational Transforms (OTs).

  • @tangle/reduce - module that takes a collection of nodes, and a "strategy" for reducing them, spits out a result
  • @tangle/graph - helper module to make preparing a collection of node into an Object which is easy to query, traverse, manipulate
  • @tangle/strategy - a module for building "meta-strategies" so that you can reduce an object composed of many different fields
    • @tangle/overwrite - most common strategy
    • @tangle/simple-set - track inclusion/ exclusion in a set, but doesn't remember "when" those happened
    • @tangle/complex-set - tracks inclusion/ exclusion in a set, as well as when something was added/ removed. (useful for authors)

These modules are what the scuttlebutt specific ssb-crut (and ssb-crut-authors) are built with

@mix.exe %w0FrKEQpzN2TpyWcxHBSXgDT+5um1ihRebnEU46QhnE=.sha256

ssb-profile v5.0.0

this splits out more types of profile that we're using in #ahau

Doing some major learning about these database schemas... basically better to start more specific, because if you overload a type it's really hard and annoying to migrate.

@mix.exe %SFQ+chk1RtGWR3pCOLlIOEMvoPlW+GOeAacoK0rD7XA=.sha256

Worked on some mermaid diagrams for the Subgroup Spec. (If you've not seen mermaid, it's a beautiful markdown tool, supported by hackmd and gitlab)



@mix.exe %KdRMc3TvjkB4QfhS5Ba41x3a/OYrlAvUGfeiiED3oFg=.sha256


@Rabble %qd9InuvhaWhMu1IEbVtx13dj3VtvKKEISBX2sXDO9rc=.sha256

This is really nice. It feels to me that this, and the associated private group spec, could be a really nice addition to the protocol guide. Either as part of the guide or as a separate document in the same style.

I've found the group spec a bit difficult to share / explain to people.

@mix.exe %QlDlWY/4QyTuqjsm1D8EoWE3gXp5wfPMRejuZQ9S/DE=.sha256
Voted This is really nice. It feels to me that this, and the associated [private
@mix.exe %duqQ4YBnmIsiojso/2v31+DdTxUiB6FlT0iHYoGnIQ0=.sha256

yep, I would love to add these diagrams to the private-group-spec @Rabble. I wonder if github supports them (groan).

The private-group-spec is pretty dense, and sits on the envelope-spec which is also dense. Who have you been wanting to share it with that found it hard?

@mix.exe %AahKXVqPREXhRo381ZbESBgiWyimqfpzF63uU0Dhkac=.sha256

We're working on kaitiaki (guardian/ custodian) subgroups. Will behave like an admin subgroup. Purpose of this group is to:

  • receive applications to join the group (via P.O. Box) and add people to the group. (note applications may contain very personal info)
  • hold certain records which are meant only for the author + kaitiaki (e.g. a very personal story, or a record of an adoption link)


@mix.exe %Oz0v1TWnwk39pUTQ554pe57lV32VOL7MvdeDUBbRZNQ=.sha256


We've been using this pattern extensively, and have been meaning to write it down :


@mix.exe %TCf7ZHmsDVVCHw1y6ZvxSVmc8dOF/ZiNNLAGA5NA3cM=.sha256


a database migration tool similar to those found in the Rails ecosystem.
Migrations are tracked in your ssb folder (using a JSON file), and only ones that have not been performed are run.

Say you have a folder of migrations you want to ensure have been performed:

├── 1632693462345-add-kaitiaki-profiles.js
└── 1632697460141-add-kaitiaki-subgroups.js

Where each migration looks like:

modules.exports = {
  up (ssb, misc, cb) {
    // does the migration
    cb(null) // make sure you remember to callback


const Migrate = require('ssb-migrate')

const migrate = new Migrate(ssb, pathToMigrationsFolder) => {
  // ..

cc @staltz @arj

@mix.exe %VkcdtQqip1eJFniEsfvXMi/P2j8RvNDUg3gW/HtHUno=.sha256

Mermaid learning

comments, line breaks, link chains!

hackmd screenshot for syntax-highlighting

Renders as:


@mix.exe %0UiLKr4FJBgnIOWLVdnj0Mok1swCMq8a3fiBANp71ig=.sha256

Added more tests and cute logging to ssb-migrate

as a migration is running, you see little pips appear each time a msg is written to your database. this is to help you know whether action is still happening while you wait

on completion you get nice ticks, and a summary of how long it took

@mix.exe %IpdpoZpEdfiGP/hNJvaka9azrBREf9AjysC86UkO8x8=.sha256


here's a gif grabbed with Peek (which happily works with regolith i3!)

@Anders %8Tin+dtYx25/2cLn47Kt3nTW9BEQ8/9THGcAZEp1MP4=.sha256

The link to the repo ssb-migrate 😏

How do you keep of state of what has been applied? And does a migration need to be idempotent?

@mix.exe %JMmMjnEjbrwme9tOMDw57fEWXh17+28DAsHAo7+Tai0=.sha256

@arj we have a file which records migrations that have been successfully completed!
Idempotent? as can be run multiple times and subsequent runs have no effect? not necessarily, because ssb-migrate makes sure it's only run once for a particular ssb db

@mix.exe %/WHWpd4BXx05Ny4v7dXKICmaNjTlNSN4TdeDowFuFoE=.sha256

more mermaid diagrams!



@mix.exe %Dwuupzqm709HVg8I2CYPrxBvhKiReaT37Bypq2zVvkk=.sha256

scuttle-testbot 1.8.0

I did some work on the logging of the replicate and connect helper functions. The default is now easier to read (to me!) and part of that is due to colouring that is unique to identities, which makes it easier to track movement of data across parts of tests


I've also added the ability to add a custom logger, or disable logging if you hate it (cc @staltz!)

@mix.exe %gZ12HWt8+384ujslYouAtcxHf4HpFdPh7y6n/JXi/7s=.sha256

problem: you're using tape and accidentally leave test.only on a test, meaning your test suite is only one test suite... which CI loves

solution: add a test which fails if test.only is found in your code!

  scripts: {
    test: "npm run test:js && npm run test:only && npm run lint",
    test:js "tape 'test/**/*.test.js' | tap-spec",
    test:only: "if grep -r --exclude-dir=node_modules --exclude-dir=.git --color 'test\\.only' .; then exit 1; fi",
    lint: "standard --fix"

cc @arj @staltz @cel

@mix.exe %7mKfaK3vTZ8jD3W2oH49W/vEqc+GY4KdAFoXV6IGwUc=.sha256

Ahau + ssb-db2

Just got off a fucking rad call with @arj. We ran through the box2/ private groups implementation in jitdb, ssb-db2, ssb-db2-box2. Most relevant to #ahau and maori data-sovereignty, it looks like it's gonna be straightforward to migrate it so that each private group (whanau/ hapu/ iwi) can use a different private sub-feed, such we can guarantee that people are only replicating family data with family members.

Hold up... wonder how that will work with pubs... hmm, to discuss

@mix.exe %PvcECSFTUcyaAXN/V9FXW2lCcgxRgN9j0ppda33cl+A=.sha256

@arj here's what I've found ahau is using sofar which we'll need to migrate. As expected, it's mostly ssb-tribes functionality:


  • mutations

    • create
      • publishes a group/init (includes groupKey), persists groupId, groupKey, authors
      • can also mint a poBox for the group, publishing a group/po-box msg
    • subtribe.create - creates a new group and links it to another with a link/group-group/subgroup msg
    • invite - publishes a group/add-member message
    • link.create - links your feed to a group with link/feed-group msg to set up a "personal group"
  • queries (use

    • list - list all tribes you have keys for
    • poBox.get - looks up the current poBoxId connected to a groupId
    • findSubGroupLinks - look up links from a group to subgroup(s)
    • subtribe.findParentGroupLinks - look up links from a group to parentGroup(s)
    • findByFeedId - look up groups that have been linked to a feedId
  • encryption

    • box/unbox P.O. Box messages (the new DM on the block)
  • listeners (use ssb.messagesByType)

    • new group/add-member msgs - triggers groupId/Key registration, rebuild
    • new group/po-box msgs - triggers poBoxId/Key registration, rebuild

ssb-ahau (graphql api)

  • using to list all sorts of records
  • listener on new messages to invalidate record cache
  • using ssb.status (or progress) to display in UI if rebulding/ indexing for a long time


  • find method contains a flumeview-search index over profile preferredName, altNames fields
@Anders %VDtyZuAaeujePUKRnYl9ckorlMc9qRLOACFwBhMteoE=.sha256

Hold up… wonder how that will work with pubs… hmm, to discuss

One way this could work is to have 2 keys for a group. One for the replication peers and one for the participants. This way a pub or backup node could be in the replication peers (and see the metafeed message), but not in the participants (and decrypt the feed contents).

@mix.exe %0Sgi2uBLpaQGWheu2VDFgJaaEOo10eJF9oPNCsAeMcs=.sha256

snapshot of a gource animated visualisation of ahau development over the last 2 years

This is a hobby project I've been working on a little recently. It's built on gource, and uses node scripting to clone all the ahau repos and combine their git log histories in such a way that they're shown relatively on the same graph.

Tonight I to a first stable version, which involved learning about node's exec and spawn and bash's sed (which I wish I'd learnt earlier! actually pretty accessible). Next up is a whole lot of tuning and visual polish


@mix.exe %o5Q9EsHF8Lwim8F6NrVrJxYcgP1wDmRaDMf/rk0Ot/w=.sha256


@mix.exe %llNqi94iyDe+lN682fL39YqKgK4MQWHEa5t/QYEbVHY=.sha256

adding the frontend!


@mix.exe %49pNpFqYCpurib73+5/1+vtDzqdRha+WfjrWhpaw2+Y=.sha256

ssb-crut demo


I recorded a 9-minute intro to ssb-crut for the show-and-tell @cblgh organised!

@mix.exe %z0Zi9nc695X9G2wSoUnzw8qMozt6UCiKJWWz85v3N04=.sha256


video of ahau's codebase till October 2021

User has not chosen to be hosted publicly
@mix.exe %UsMXFSJkc6AKWOS7S1EPgn9ZsMLHrrinDcPgFb0UBDQ=.sha256

@skyebend that was us extracting a module into it's own repo. The same content is later re-introduced to this graph but in a much more trimmed down version

@mix.exe %tK8gDtSK8LKEAvxcg6z8EeW75MeEPsT7ceJFB8S+dQ0=.sha256

We're starting to support rending graphs which allow you to choose where to put an adopted child... as in display them closest to the family which is most appropriate to your subjective perspective


Here Alice had a birth-child AAA2, but they were whangai'd (like adopted, but often given within a whanau / family) to Carol. For this family they want to show AAA2's most important relationship as being with Carol (other families may choose differently)

The code is getting pretty interesting

@mix.exe %YnSCKTgbCcYIKsP04KlvdEvOfQXJVp70s360rM6pUC0=.sha256

Here's a nice rendering

@mix.exe %B7R9Ax5UJ5xpkFHkgE9IED4OntbsL3F+uvQWwkhXqjk=.sha256

Worked on better logging:

@mix.exe %e+4xmr+3s5g1JBfLj8Wv1tcRTYrkznuh6AR7XVfpTJ0=.sha256

Wrote up some specs today:

whakapapa spec

This defines how messages are brought together to weave a tree. Note the link messages which behave kinda like a JOIN table, allowing us to store additional meta-data (like is this a birth child or an adopted child?), as well as have easy ability to know who authored the link, and to point and and comment/ attest/ verify/ refute the link

@mix.exe %tF4GXbiNO0RZ611GjlT19ZeeOJaScnSxQEnedOaPaH8=.sha256

Colours from : (wow I love designer chosen palettes :heart:)

@mix.exe %erpLHCHjUGnVpVuFthyvEMYg1aAQoCIcuIUCmT58KoE=.sha256

Meanwhile @cherese has been working on a spec that allows communities to dynamically add custom fields they want profiles in their tribe to support. Database migrations on the fly in a p2p database... easy :S


@mix.exe %WfDM/6Kjgwd5rOm65PZ2t93jvCwefslqNilnZ8iZijk=.sha256

Check out my 2022 dev-diary here: %+QNrql0...

Join Scuttlebutt now