Mix's dev diary 2021

It's been a while since I posted about what I've been building in any focused way. I've been inspired by the NGI teams posts to revisit this practice. I also think it's important because many of you likely don't see the scuttlebutt work that's being done with #ahau so clearly, and there's some great things coming out of it.

I might post a summary of the rest of 2021 at some point, but to get underway I'm gonna start posting from the present

#dev-diaries #dev-diary
art by Raphael Lacoste

ssb-keyring

A JS library extracted from ssb-tribes which is useful for persisting and synchronously querying for encryption keys.

https://gitlab.com/ahau/ssb-keyring

Extracting it was following a pattern @arj started by extracting ssb-private-group-keys/. It feel really good to give this module some space .... I've expanded it into clearer domains and broken the tests out to match those domains.

It was also motivated by the drive to add "P.O. Box" keys to this store without it becoming a total mess. This is part of a bigger epic for #ahau - Subgroups. Our first feature that we're driving towards is kaitiaki-only subgroups, with the added functionality:

• people can send messages to the kaitiaki (without being part of their group)
• people can share maintenance of personal details (e.g. home address) with kaitiaki
  • this is a feature I'm pretty excited about. The intent is this offers privacy from the general group AND transparency so data you've shared is still "yours"

definitions

• kaitiaki - guardian

ssb-crut v3.1.0

Adds a new feature which allows you to attach a record tangle to an arbitrary root, for example attaching a settings record to a gathering, or group

// specify the behaviour of our "settings" record
const spec = {
  type: 'settings',
  arbitraryRoot: true,  // << NEW!
  props: {
    admins: require('@tangle/simple-set')()
  }
}
const settings = new CRUT(ssb, spec)

// update the settings record attached to a gathering
const updateId = await settings.update(gatheringId, {
  admins: { add: '@mixoxcviuoiu...ed25519' }
})

// or a group I'm a member of!
const updateId2 = await settings.updateGroup(groupId, {
  admins: { add: '@mixoxcviuoiu...ed25519' }
})

CRUT supports promise + callback style thanks to @Luandro Pàtwy

We're looking to use this in #ahau because it an elegant way to e.g. register the poBox of a group and attach it in really easy to look up way.

ssb-crut v3.2.0

Adds support for the new ssb-db2 and it's indexes (instead of ssb-db and ssb-backlinks).
Thanks to @arj for this new functionality!

ssb-tribes v2.3.0 - subtribes!

This adds the ability to mint private groups which are "inside" another group:

in practice just another group EXCEPT a link/group-subgroup message is published in the parent group pointing to the existense of a subgroup

We've also added the ability to mint groups with a P.O. Box, or to add one later

• ssb.tribes.create({ addPOBox: true }, cb)
• ssb.tribes.addPOBox(groupId, cb)

both of these publish a group/po-box message to the group, which transmits the P.O. Box keys to other members of the group so that they can use open private messages sent to the group from people outside the group

Example usecase

In ahau we are starting all groups with a "kaitiaki subgroup" - this group will be responsible for approving applications to join the group, and allows application details (and any discourse) to be private to the kaitiaki and applicant.

After an applicant is approved, they will be added to the main group, but the approving kaitiaki will also mint a "kaitiaki only" version of the profile (which the person who it's about will also have read/ write access to), allowing safe storage of contact details in a place not public to the whole group.

@colin (an academic helping #ahau out) is working on merging tangle branches, which is a surprisingly tricky problem! We noticed that writing tests was a grind, so I copied the Mermaid format for flowcharts, and made a little tool!

@tangle/test

To build a graph of nodes like:

   A
  / \
 B   C
  \ /
   D

assuming direction is top->bottom

const { buildGraph } = require('@tangle/test')

const nodes = buildGraph(`
  A-->B[cat]
  A-->C
  B-->D
  C-->D
`)

// => {
//   A: {
//     key: 'A',
//     previous: null
//   },
//   B: {
//     key: 'B',
//     value: 'cat',
//     previous: ['A']
//   },
//   C: {
//     key: 'C',
//     previous: ['A']
//   },
//   D: {
//     key: 'D',
//     previous: ['B', 'C']
//   }
// }

More context: the @tangle/* repos are generic tools for reducing Directs Acyclic Graphs (DAGs) which contain Operational Transforms (OTs).

• @tangle/reduce - module that takes a collection of nodes, and a "strategy" for reducing them, spits out a result
• @tangle/graph - helper module to make preparing a collection of node into an Object which is easy to query, traverse, manipulate
• @tangle/strategy - a module for building "meta-strategies" so that you can reduce an object composed of many different fields
  • @tangle/overwrite - most common strategy
  • @tangle/simple-set - track inclusion/ exclusion in a set, but doesn't remember "when" those happened
  • @tangle/complex-set - tracks inclusion/ exclusion in a set, as well as when something was added/ removed. (useful for authors)

These modules are what the scuttlebutt specific ssb-crut (and ssb-crut-authors) are built with

ssb-profile v5.0.0

this splits out more types of profile that we're using in #ahau

Doing some major learning about these database schemas... basically better to start more specific, because if you overload a type it's really hard and annoying to migrate.

Worked on some mermaid diagrams for the Subgroup Spec. (If you've not seen mermaid, it's a beautiful markdown tool, supported by hackmd and gitlab)

src: https://gitlab.com/ahau/spec/private-subgroup-spec

This is really nice. It feels to me that this, and the associated private group spec, could be a really nice addition to the protocol guide. Either as part of the guide or as a separate document in the same style.

I've found the group spec a bit difficult to share / explain to people.

yep, I would love to add these diagrams to the private-group-spec @Rabble. I wonder if github supports them (groan).

The private-group-spec is pretty dense, and sits on the envelope-spec which is also dense. Who have you been wanting to share it with that found it hard?

We're working on kaitiaki (guardian/ custodian) subgroups. Will behave like an admin subgroup. Purpose of this group is to:

• receive applications to join the group (via P.O. Box) and add people to the group. (note applications may contain very personal info)
• hold certain records which are meant only for the author + kaitiaki (e.g. a very personal story, or a record of an adoption link)

ssb-link-spec

We've been using this pattern extensively, and have been meaning to write it down :
https://gitlab.com/ahau/spec/record-link-spec

ssb-migrate

a database migration tool similar to those found in the Rails ecosystem.
Migrations are tracked in your ssb folder (using a JSON file), and only ones that have not been performed are run.

Say you have a folder of migrations you want to ensure have been performed:

migrations
├── 1632693462345-add-kaitiaki-profiles.js
└── 1632697460141-add-kaitiaki-subgroups.js

Where each migration looks like:

modules.exports = {
  up (ssb, misc, cb) {
    // does the migration
    cb(null) // make sure you remember to callback
  }
}

Then:

const Migrate = require('ssb-migrate')

const migrate = new Migrate(ssb, pathToMigrationsFolder)

migrate.run(err => {
  // ..
})

cc @staltz @arj

Mermaid learning

comments, line breaks, link chains!

hackmd screenshot for syntax-highlighting

Renders as:

Added more tests and cute logging to ssb-migrate

as a migration is running, you see little pips appear each time a msg is written to your database. this is to help you know whether action is still happening while you wait

on completion you get nice ticks, and a summary of how long it took

here's a gif grabbed with Peek (which happily works with regolith i3!)

The link to the repo ssb-migrate 😏

How do you keep of state of what has been applied? And does a migration need to be idempotent?

@arj we have a file which records migrations that have been successfully completed!
Idempotent? as can be run multiple times and subsequent runs have no effect? not necessarily, because ssb-migrate makes sure it's only run once for a particular ssb db

more mermaid diagrams!

source: https://gitlab.com/ahau/spec/private-subgroup-spec/-/blob/main/GROUP-APPLICATION.md

scuttle-testbot 1.8.0

I did some work on the logging of the replicate and connect helper functions. The default is now easier to read (to me!) and part of that is due to colouring that is unique to identities, which makes it easier to track movement of data across parts of tests

I've also added the ability to add a custom logger, or disable logging if you hate it (cc @staltz!)

problem: you're using tape and accidentally leave test.only on a test, meaning your test suite is only one test suite... which CI loves

solution: add a test which fails if test.only is found in your code!

{
  scripts: {
    test: "npm run test:js && npm run test:only && npm run lint",
    test:js "tape 'test/**/*.test.js' | tap-spec",
    test:only: "if grep -r --exclude-dir=node_modules --exclude-dir=.git --color 'test\\.only' .; then exit 1; fi",
    lint: "standard --fix"
  }
}

cc @arj @staltz @cel

Ahau + ssb-db2

Just got off a fucking rad call with @arj. We ran through the box2/ private groups implementation in jitdb, ssb-db2, ssb-db2-box2. Most relevant to #ahau and maori data-sovereignty, it looks like it's gonna be straightforward to migrate it so that each private group (whanau/ hapu/ iwi) can use a different private sub-feed, such we can guarantee that people are only replicating family data with family members.

Hold up... wonder how that will work with pubs... hmm, to discuss

@arj here's what I've found ahau is using sofar which we'll need to migrate. As expected, it's mostly ssb-tribes functionality:

ssb-tribe

• mutations

  • create
    • publishes a group/init (includes groupKey), persists groupId, groupKey, authors
    • can also mint a poBox for the group, publishing a group/po-box msg
  • subtribe.create - creates a new group and links it to another with a link/group-group/subgroup msg
  • invite - publishes a group/add-member message
  • link.create - links your feed to a group with link/feed-group msg to set up a "personal group"

• queries (use ssb.query.read)

  • list - list all tribes you have keys for
  • poBox.get - looks up the current poBoxId connected to a groupId
  • findSubGroupLinks - look up links from a group to subgroup(s)
  • subtribe.findParentGroupLinks - look up links from a group to parentGroup(s)
  • findByFeedId - look up groups that have been linked to a feedId

• encryption

  • box/unbox P.O. Box messages (the new DM on the block)

• listeners (use ssb.messagesByType)

  • new group/add-member msgs - triggers groupId/Key registration, rebuild
  • new group/po-box msgs - triggers poBoxId/Key registration, rebuild

ssb-ahau (graphql api)

• using ssb.query.read to list all sorts of records
• listener on new messages to invalidate record cache
• using ssb.status (or progress) to display in UI if rebulding/ indexing for a long time

ssb-profile

• find method contains a flumeview-search index over profile preferredName, altNames fields

Hold up… wonder how that will work with pubs… hmm, to discuss

One way this could work is to have 2 keys for a group. One for the replication peers and one for the participants. This way a pub or backup node could be in the replication peers (and see the metafeed message), but not in the participants (and decrypt the feed contents).

snapshot of a gource animated visualisation of ahau development over the last 2 years

This is a hobby project I've been working on a little recently. It's built on gource, and uses node scripting to clone all the ahau repos and combine their git log histories in such a way that they're shown relatively on the same graph.

Tonight I to a first stable version, which involved learning about node's exec and spawn and bash's sed (which I wish I'd learnt earlier! actually pretty accessible). Next up is a whole lot of tuning and visual polish

src: http://gitlab.com:ahau/prototypes/ahau-code-viz

woops: http://gitlab.com/ahau/prototypes/ahau-code-viz

adding the frontend!

ssb-crut demo

https://www.youtube.com/watch?v=URIUbupxxU4

I recorded a 9-minute intro to ssb-crut for the show-and-tell @cblgh organised!

https://www.youtube.com/watch?v=Ep1qI00qOQQ

video of ahau's codebase till October 2021

@skyebend that was us extracting a module into it's own repo. The same content is later re-introduced to this graph but in a much more trimmed down version

We're starting to support rending graphs which allow you to choose where to put an adopted child... as in display them closest to the family which is most appropriate to your subjective perspective

Here Alice had a birth-child AAA2, but they were whangai'd (like adopted, but often given within a whanau / family) to Carol. For this family they want to show AAA2's most important relationship as being with Carol (other families may choose differently)

The code is getting pretty interesting

Here's a nice rendering

Worked on better logging:

• color-tag module
• scuttle-testbot colorLog for message logging

Wrote up some specs today:

• Documenting ssb threads
• Whakapapa Spec (gitlab)

This defines how messages are brought together to weave a tree. Note the link messages which behave kinda like a JOIN table, allowing us to store additional meta-data (like is this a birth child or an adopted child?), as well as have easy ability to know who authored the link, and to point and and comment/ attest/ verify/ refute the link

Colours from : https://colorpalettes.net/color-palette-4308/ (wow I love designer chosen palettes )

Meanwhile @cherese has been working on a spec that allows communities to dynamically add custom fields they want profiles in their tribe to support. Database migrations on the fly in a p2p database... easy :S

Check out my 2022 dev-diary here: %+QNrql0...