You are reading content from Scuttlebutt
@andrestaltz %TIddJIPkn0JmTQG/Ao7gMiSsaVhzx1QC3+QDLuQnhFQ=.sha256

Team Diary

NGI Assure group ("Private Groups in Manyverse")

a screenshot of a Jitsi call showing five participants smiling: Mix, arj, staltz, Jacob, Nicholas

@Mix @arj @andrestaltz📱 @Powersource @nonlinear

#bat-butts #batts #manyverse #private-groups

I realized that people in the SSB community may not know what we're working on, so in an attempt to bring transparency to our work and document our journey, I'm starting this thread that will summarize the whole team's weekly progress. Not quite a dev diary, because we'll also include design and other aspects along the way.


We started by trying to find a time in common for a video call. This proved to be quite hard! We have people in US East Coast, New Zealand, and Europe, and if you take a look at timezones, this means one of us will inevitably be in their usual "sleep" time range. We found a solution when nonlinear volunteered to wake up at 5AM on Fridays for the video call! For Mix, that's 9PM. In Europe, that's roughly lunch time. Our first team meeting was on 20th of May.

Now that we have the basics of teamwork set up (our communication channels are on Signal, our project management is on Airtable, our notes are on Hackmd), we can finally start work on core work.

If you peeked at this diagram for SSB work in the coming months you'll notice that we have 5 boxes inside the "funded by NGIa" rectangle. Those are the 5 milestones we have written down in our grant document. Once those are completed, we'll be paid by NLnet. They are:

(1) User experience design of groups

We want to study user needs and design the user interface of private groups in such a way that is consistent with Manyverse’s current workflows. Nicholas Frota, a product designer by profession, will lead this phase.

(2) Private chats extended to unlimited members

Manyverse currently has private group chats, limited to a maximum of 7 participants. We want to introduce “SSB Private Groups” as the engine behind group chats, extending the participant count from 7 to (theoretically) unlimited. This will allow us to test the waters of “SSB Private Groups” while not making user interface changes, yet.

(3) Removal of members by moderators

An important property of groups is community safety where boundaries are set, such that infringers are accountable for their actions and/or removed from the group. “SSB Private Groups” as a subprotocol does not yet take removal into consideration, so we want to design new rules in the protocol to allow groups to effectively exclude infringers, one way or another.

(4) Space-efficient replication of group messages

Group content is encryted to only the members of a group, and are irrelevant to group outsiders. While SSB stores replica of contents from friends, it is undesired to store encrypted content from friends if that content can never be decrypted. We need to find ways of utilizing peer-to-peer storage efficiently and respectfully.

(5) Groups as a new feature in Manyverse

Finally, we want to put together all the components of the previously mentioned milestones and deliver new group functionality in Manyverse that enables hundreds of members per group, member removal, seamless replication of group content, via an easily understandable user interface.

It seems like these are ordered linearly, but actually we started with (1) and (4). Turns out that (2) depends on (4), so (4) is really important.

Mix, arj and I had a video call to give Mix a primer on metafeeds, and to sketch a first draft of how would private groups replicate as metafeeds and subfeeds. We have a nice sketch, but it has some open questions that need answers before implementing. Mix also got acquainted with ssb-meta-feeds, giving some feedback on its API, and a README PR.

Nonlinear got busy with (1), and the design process starts by mapping out all the assumptions (be them technical, sociotechnical, or social) surrounding Groups (we're dropping the word "private" by the way). Notes can be found in this GitLab issue, but these notes will evolve a lot over time. So if you're reading this in the distant future, I don't know what you'll see there. :) We had a couple of meetings to discuss these assumptions, and we noticed how much nonlinear's design work is extremely important!

One of the conclusions that we arrived was that we cannot promise that Groups are for the security-minded folks. These SSB Groups can't even be on the level of Signal chats in terms of privacy and security. This has to do with limitations in the cryptographic schemes we're using. Any group member can leak the symmetric key of the group to anyone else, which means those others could not only read one message, they could read all messages! And they can also write new posts such that existing members can read them (unless we stop that on the UI layer). It's important that we acknowledge this and that we don't promise too much.

That said, the cryptographic schemes we'll use are an improvement over the wide-open Public posts, because Groups can have some kinds of boundaries that Public cannot. We're reserving "Secure Groups" (with double ratchet-style perfect forward secrecy and other stronger properties) as future work, where we'll be able to cater for personas that need strict security guarantees. Communicating the "security"/"insecurity" of private-groups-based Groups is going to be an important task. At this point I'm very grateful we have a professional designer who helps map the assumptions, concepts, personas, limitations, and utilities.

(continues in the next post...)

@andrestaltz %l9cDEJdATQnCP0QxgeZFdYZGd13hC1igJX3YTjqJ1pU=.sha256

Milestone (2) is also about putting the private-groups libraries in Manyverse and making them "work", or at least not crash. We can't complete (2) without finishing (4), but we can get started with (2) at least. So I started doing that.

I took a look at ssb-tribes and thought of putting it in Manyverse. Turns out that ssb-tribes doesn't support ssb-db2 yet. So that led me to look at ssb-db2-box2 instead, which is a module arj made while #ssb-ngi-pointer was active. This led me to think of how nice it was that ssb-db supported addBoxer and addUnboxer so you could easily choose between box1 or box2. In parallel, I'm taking a look at arj's buttwoo feed format. This all disorganized, and I think it would be hard to work with these modules in a way that minimizes bugs and confusion.

Just look at this mermaid diagram of modules involved:

graph diagram with boxes and arrows, where boxes are npm modules and arrows are relationships of dependencies

As you can see, it started as small investigation of how to put ssb-tribes in Manyverse, and it ended with me investigate deep rabbit holes. ssb-db2 is ripe for a refactor. We need a seamless way of adding new feed formats, crossing those with encryption formats, etc. There are 3 dimensions involved: feed format, encryption format, and serialization format. I began working on a refactor of ssb-db2 and ssb-db2-box, which kept expanding to include refactoring other modules, like ssb-bendy-butt, ssb-keys, ssb-ebt, and ssb-validate.

My notes for this started looking crazy:

notepad open up with several notes written down about ssb-db2 and encoding formats

On Friday, this along all the rest of the work, felt overwhelming. I mean, last week I shared how the coming months are going to be overwhelming, and then I figure out this massive refactor to be done. It got me worried. Are we going to manage doing all this?

But over the weekend, I took some 2 hours to poke at this refactor problem again, and it slowly started to make sense. Today, Monday, I showed the idea to Mix and Arj and they gave green light. Basically the plan is to add addFeedFormat() and addEncryptionFormat() to ssb-db2, and to separate these two as orthogonal dimensions. The devil is in the details, but I feel like those details have designated places in the refactor plan, and it's mostly about executing the refactoring (and, well... later figuring out bugs and corner cases, the big unknowns that have the chance of ruining the plan).

This refactor reminds me of the bermuda triangle refactor, except this time it feels 10x bigger than bermuda. Oh well!

As closing notes, last week I was browsing the "original notes" document Dominic and Keks and Mix (?) wrote for private-groups, and stumbled upon this harsh reminder of the nature of our work:

Some positions are not attainable within the scuttlebutt architecture, due to interactions between hard problems in cryptography and hard problems in distributed systems.

:sweat_smile:

@SoapDog (Macbook Air M1) %VbXecaflAot4W5/iMn+FO11D3+znSH/487MQF3ot8TA=.sha256

Folks lost the chance to call it the NGI Private Butts group....

User has not chosen to be hosted publicly
@The System %SzTs5llquVzUJ5IpL9sQieFtZhkDXb8x42YJGmlwl90=.sha256

E: Good luck!

@luandro %V0Fski4d7yUBB/peY9QOd0pG++Cb1yxQoHaSTLk3klM=.sha256

Awesome to be able to follow thoughts and progress, thanks for this.

@andrestaltz %MJMhTHanQEj9A79uoMf2OXkSZ8XcknAh6AZToZ8oOag=.sha256

Update

After additional meetings Nicholas had with Mix and me, the Assumption Mapping is complete! We now know the scope of Groups, such as the technical limitations, UI requirements and data requirements, the target audiences, as well as what is out of scope. The next steps for design are to study and design the flows that users will go through when joining and interacting with groups.

On the technical side, Mix and Arj made a first draft for the structure of metafeeds and subfeeds to support groups to replicate. They made a diagram and wrote down the step-by-step pseudocode for Alice to invite Bob to a Group.

Diagram showing a root metafeed pointing to a groupsRoot sub-metafeed

On the coding side, I've been continuing the refactor of ssb-db2. I also made improvements to ssb-keys and bipf, adding new helper APIs to those. It's still a lot of mess to untangle, and although I'm making progress, I'm not sure it's the best design. One constraint that we settled on is to not support lipmaa links, because those require a whole lot of "looking backwards" logic that will make the code more complex. So bamboo is ruled out, for now. I'm just trying to find a sane structure of adding feed formats, and the scope currently is classic, bendybutt-v1, and buttwoo-v1. The feed formats you'll be able to add are many, but they have to follow some strict requirements.

@andrestaltz %mzzvNjsYE3UqZpYvjYJuSVvJk1C2lNX6s7KbAi09Qr4=.sha256

Update

Last week was good, in terms of coding. I finished the nasty parts of the ssb-db2 refactor and now it's a downhill ride. I'm quite happy about this refactor, because in many ways it simplifies code, and helps to know where to put new code. This separation of concerns also helped me understand how box2 key management should work. Arj and I also tweaked the performance of buttwoo a bit. We knew the refactor would cause a performance regression, so we tried to recover some performance without undoing the refactor, and I think we got to a satisfactory level of performance. When everything is done, the structure of modules should look like this:

mermaidjs diagram of a computer-science-graph of npm modules related to ssb-db2

Mix, Arj and I also continued tweaking the design of metafeeds to accommodate groups. The current state is that under your root metafeed you'll have a predefined number of subfeeds, only these: main, indexes, groups, social, games, collab. And then under each of these we will have a recommendation to use first-byte-categorization (I don't know what to call it, it's similar to what ssb-blobs does to your ssb/blobs/sha256 folder). This is to avoid any metafeed having too many messages to be replicated, with "too many" meaning "thousands". With first-byte-categorization we limit the message count of a metafeed to 256. More details in issue ssb-meta-feeds-spec#30, and the diagram below:

mermaidjs diagram of a computer-science-tree of metafeeds and subfeeds

Nicholas and Mix collaborated on making diagrams to describe the invite flow, to add new members. We didn't have our Friday meeting last week, so they'll show these diagrams this week. I can update y'all about these diagrams on the next Team Diary update. :v:

User has not chosen to be hosted publicly
@andrestaltz %j0qbBhOvj3BSKP3OmCKZjcCp4IbvyWarzzsiE5Gv8jg=.sha256

Update

On Wednesday, Nicholas, Mix and I had a short meeting to review the "registration flow", which means the process of a person discovering about a Group and applying to join the room.

A group's conversations are private to its members, but the existence of a group is not private. Every group will have what we call a "door", which is a profile detailing what that group is about, who is a member, and instructions to join the group. Any person can then apply to join that group, answering some questions to prove they should be a member. The admins will then see this and have the power to approve or reject the application. This is not the only way of joining a group, the other way is via an invite. This flow is for the random people who are interested in joining a group. See diagram below:

FigJam flowchart detailing how people can apply to become member of any group

On another topic, I'm almost done with the "downhill" part of the ssb-db2 refactor. I created these modules:

@andrestaltz %KErqVQtte1dTA+sxjkKp8T63AdDHxNX448bATE6r09E=.sha256

Update

The ssb-db2 refactor is done and just waiting for changes to ssb-keyring and ssb-db2-box2 before we can merge it and release.

Speaking of those two modules, it's nice that the refactor is done and we can focus on "actual" box2 work, which in this case is generalizing ssb-keyring so that it does key management, but allowing keys to always be passed in. This way, other modules can take care of creating or discovering the encryption keys. Our plan is to either derive the encryption keys from metafeed seeds, or to "discover" each group's keys via the invite messages. Anyway, all the keys discovered will be put in ssb-keyring, and ssb-db2-box has to be able to control/manage ssb-keyring. Mix and I have been working on that. Mix has this PR on ssb-keyring and I have this PR on ssb-db2-box2.

In last week's team meeting we also discussed bigger picture ideas, mostly relating to design. We discussed what should happen when you join a group that contains a person whom you block. Instead of ignoring/hiding that person's content, we chose to show their content and to warn you of the presence of blocked person as soon as you join the group. Showing their content is important to preserve shared context for that group. Depending on the type of group, you might engage with the blocked person in a very different way you would on the Public side. E.g. imagine a group for a work/professional environment, where it doesn't make sense to split the conversation/context based on dislikes between people. On the other hand, on the Public side you have different expectations of who you want to engage with, you are more free to shape the context. In a sense, Groups are Context, Code of Conduct, and the maintenance of these.

One thing I learned in last meeting was that random people will not be able to see who are the members of a group. We spoke briefly about how that will happen in practice in the design of metafeeds and subfeeds.

Another thing we discussed in our team chat was a way for random people to apply to become a member of a group: we could just allow the person to start a private chat with the admins. But this would require showing to the public who the admins of any group are. Instead, Mix suggested that we just use P.O. Box, which is a way of encrypting something to a subset of the group's members (in this case, the admins). So the applicant sends a private chat message to the admins, not knowing who they are, and when the admins reply then they reveal themselves to the applicant, but they're not revealing themselves to the public. That's nice.

User has not chosen to be hosted publicly
@andrestaltz %QxDZu7Y1Dk3WSU6h8NsxvsqXhKdsc6nOlyASi2RqxcQ=.sha256

Update

On the coding side, not much has changed: Mix and I have been working on ssb-keyring for ssb-box2. We didn't have a lot of time last week because I was focused on a Manyverse release, and Mix... with the cat.

But we had a productive team meeting, nicholas+me+arj where we discussed a couple of tricky topics, such as:

Groups as "persons"

Nicholas is more and more convinced that groups are "identities" in a way much similar to how persons have identities. This would be reflected on presentational aspects in the app, and in capacities ("groups should be able to publish public posts"). We discussed the tricky limits of this abstraction though, like:

  • can groups be blocked? (we reached a conclusion that "yes", but we're defining what effects that blocking would cause)
  • how does a group post something on its "feed"? (there is no single author for the "feed", so we'd have fork risks)
  • if groups are persons, can I invite a group to a group? thus creating a subgroup (we decided to not allow this for the time being, otherwise we're complicating our work too much)

Closing a group

If group admins have the power to remove members, could they remove everyone from the group, essentially "deleting" the group. We didn't reach a conclusion for that, but we discovered that "leaving" a group is functionally equivalent to "blocking" the group, on the data layer. Or, there could be an exception where you unsubscribe from the group's contents, but are still willing to replicate the group locally, for the purpose of helping other peers.

Group where admins disappear

Related to "leaving" groups and stopping its replication, we also discussed about an indigenous use case, where a group contains a lot of important content that is old/generational, and should be kept forever, even if the group's admins pass away. Groups where inactivity does not imply autodeletion.

One idea I tossed, to support this, is forking a group such that a person elects themselves as the admin of an entirely new group which inherits content from another group.

Changes to SSB at the conceptual layer

So far, SSB has been a simple map from "1 feed ID to 1 person". The conceptual layer is the data layer. But this abstraction is showing its limits. On the one hand it's simple, and in many ways simple is fantastic, but on the other hand, we need to find solutions for: partial replication, stable storage, groups, same-as, etc.

One way how 1-feed-1-person already doesn't work is when you have two or more devices. That's the case for many people on SSB and we need same-as (fusion identity) to solve that.

For partial replication, we've come up with metafeeds which create a tree of feed IDs that you own, each branch for a specific purpose.

For (private-)groups, we have the group ID as a "cloaked msg ID" and increasingly we're noticing that these group IDs can function as "identities" or "persons".

So going forwards we're going to have to break apart the conceptual layer from the data layer. Feed IDs will just represent a sig-chained sequence of messages, but not much more. When thinking about a person, we're going to have create another representation for them, such that (e.g.) when addressing Alice in a DM or in a mention, the implementation will have to figure out "which of Alice's feeds do we target in this context?"

We want to be intentional about this tradeoff and willingly pay the cost of letting go of the simplicity of 1-feed-1-person.

User has not chosen to be hosted publicly
@andrestaltz %RwiLPOgVeLKQGf1UPzLZsGqP8FEn4v65BmmDzxdvn8w=.sha256

Update

Apart from finishing the ssb-db2 refactor, last week we also sat down to figure out some problems we need to solve before we solve our bigger problems.

User Agents

One of these smaller problems is "How do we know if a friendly peer supports private-groups?" and there is no simple answer to this. We thought of using the heuristic "if they have a root metafeed, then let's assume they also support private-groups", but we began thinking of how to properly solve this, and we kept talking about User Agents. Somehow, each app would publish or indicate what software stack they are using. We still have to flesh out all the details, but Mix kickstarted a repo for a spec for this: ssb-user-agent-spec.

Permissions

We discussed quite a lot whether any group member should be allowed to invite others to the group, or if only the admin can do that. We also discussed whether this should be a group setting. This problem is actually quite thorny, because it's about authority in a network that is fundamentally without authorities. But we want to address it with thought, and for that we need to write down our thoughts, so Mix made ssb-permissions-spec and ssb-group-admin-spec.

Metafeed structure for groups

This spec is just the version 0.0.1 of the designs and diagrams we've been gathering while meeting many times to discuss how should the content of groups be published among metafeeds and subfeeds: ssb-meta-feed-group-spec.

@andrestaltz %mIcIeWirM8X/aAsFHbh9Mv5kDknuuVsSMEg21cfZM1o=.sha256

Update

It's been some time! We had a summer holiday break, and other obligations (such as launching the new storage screen in Manyverse), but we're back.

We were a bit stuck with "metafeeds for groups" (see how short our spec is so far), so we had a couple of meetings to figure things out.

First, @Mix and I met and we discussed a few times the interplay between group-specific feeds and app-specific feeds. What if you have a cryptographic private group under the context of a Tic-tac-toe game, how are their feeds organized? Metafeeds dictate a strict hierarchy, a tree structure for the feeds. So we thought, do we put groups under apps? Or do we put apps under groups? We tried to predict what were the pros and cons of each approach. Neither approach felt "right".

diagrams for the tree structure of metafeeds, subapps under groups, and groups under subapps

Then, we briefly discussed putting all the subfeeds directly under the root metafeed, such that the "tree" is just very wide and not that much an actual tree. The big downside with that approach is that whoever replicates you will have to be aware of all your subfeeds, and that means a large overhead. E.g. if one app happens to be naively coded to create a new subfeed for each session, that pretty much spells the death of your root metafeed because it'll have hundreds/thousands of messages, and that's a lot of overhead for a "partial replication" strategy. Thus follows that we need to find a tree structure that allows the remote peer to replicate only parts of it.

diagram for the tree structure of metafeeds, where all subfeeds are under the root

We had a second meeting, @arj, Mix, and I and early in the meeting we explored one new idea: letting go of "domains" or any semantics for the tree structure and using only 1st-byte sharding and versioning for the tree structure. Previously, we had thought that the structure would be

root->semanticDomain1->semanticDomain2->shard->version,

notice how sharding and versioning are deep in the tree, closer to the leaf. This time, we pulled those two upwards, inverting the order, and putting them close to the root, like this:

root->version->shard->subfeed

Where the version concerns the whole tree structure, and the shard is a little bit of entropy extracted from each subfeed. The version means that in the future, if we want to design an entirely different tree structure (e.g. if we want to reinsert domains after all), we can refactor the subfeeds by putting them under a new version feed.

diagram for the tree structure of metafeeds, where there is a root, a v1 feed, 1st-byte feeds, and content feeds

The three of us got excited with this structure! It means it's less "UML Class diagram" and it's more "Merkle tree", concerning itself only with shape and bytes, not with semantics. The sharding is important so that remote peers can replicate only the parts they are interested in.

We have a PR for ssb-meta-feeds-spec that we're tweaking before merging, and then Mix and I have been studying a bit how should the shards be created. We were debating a bit whether it should be one byte (i.e. 256 total shards) or one nibble (16 total shards). For that, we wrote a pseudo-academic paper, what we call "sharding math". After running a few simulations and imagining what the first year of metafeeds in production is going to look like, we settled for 16 shards, i.e. a nibble.

Next steps from here are merging the spec PRs and then starting to code these new changes in the JS ssb-meta-feeds library!

@mix.exe %AxG0m7N6TmUEZAiWKqaa6EFfCD5xMSv3uJb/KvTihL4=.sha256
Voted ## Update It's been some time! We had a summer holiday break, and other ob
@mix.exe %gZ7xxxbUMXLbHkfUZEGad8gf2x5WsvI+TNRoIdYPXCc=.sha256

cc @Rabble @Matt Lorentz keen to talk to you about meta-feeds soon

@mix.exe %Y2yANwCCVp/gHso+b8a8M2sAXV2tKwepalCAn5a3CCA=.sha256

Started drafting the meta-groups spec today.. it hurts my brain a bit (so many little things to keep in mind)

image.png

https://github.com/ssbc/ssb-meta-feed-group-spec/pull/1

@andrestaltz %hvr4yj0oAh9D0izawiD5ygD/doN4ERM5qFxLr6h7Fms=.sha256

Update

We've got the specs in a fairly good shape, and we can start coding.

Index feeds rework. I was going to start implementing shards in ssb-meta-feeds, but I realized that a lot of its APIs were used by ssb-ebt for index feeds in a way that is not great. Arj and I found a way to refactor this, which deletes a bunch of glue code. We ended up changing the index feed spec a bit so that it uses a new "feed format" called indexed-v1, which is basically a copy of the classic feed format, but tagged with new SSB URIs like ssb:feed/indexed-v1/___. Then, a lot of shuffling around in ssb-ebt and the lower level modules like ssb-bfe, ssb-uri2, ssb-keys, etc. Finally, we also updated ssb-index-feed-writer renamed to ssb-index-feeds. This wasn't central to our work on Groups, but allowed us to delete code from ssb-meta-feeds, which is good because we'll be working a lot on that module.

sharding in ssb-meta-feeds. @Mix began working on auto-sharding in ssb-meta-feeds, and the logic is looking good in this PR. He's going to rework our tests, though, to make them more flexible to changes.

ssb-tribes2. @Powersource is back in the game, and we helped kickstart the ssb-tribes2 repo (not ready, do not use!). This is essentially ssb-tribes, but based on ssb-db2.

On a project management level, we tried to figure out what is the timeline for our first payment to come in, and that requires the first milestone to be completed. We think we'll be done with the "replication" milestone by the 1st week of November. So that gives us about a month and a half. It sounds like a suitable deadline. Sounds "close", but on the other hand we have 3 active developers, and we're already in the phase of writing code.

Join Scuttlebutt now