You are reading content from Scuttlebutt
@cel %07VekPqfjZf5QWeok8Fu7ast8qkF5KEN1i1YpKUXctc=.sha256
Content warning: ssb dev rabbit-hole

Design for installers blob web publishing.

%0/ge9m0...

Use a new message type to express mapping URL/paths to blobs. ssb-links has an index for this for link names in the mentions array, so use that. The mention name could have some prefix indicating it is for this application (like ssb-npm uses a "npm:" prefix), but if we use the fully-qualified URL, that is good enough and has a nice property of corresponding to what URL the blob should be served at.
Example:

{
  "type": "installers",
  "mentions": [
    {
      "name": "https://installers.scuttlebot.io/example/hello.sh",
      "link": "&qNy5xZ4bhu61z4k+XenbtDnt1IJp0b/gLJ/vdzd5vxM=.sha256"
    }
  ]
}

On the server side: serve the blobs as static files (e.g. using nginx), rather than having a ssb-server plugin/client serve requests. The reason is to have high availability, since ssb-server often hangs or crashes. Make a tool to sync changes from ssb messages to the file system, including a live mode. The tool should also have an option to publish blobs from the file system to ssb.

Can I get input/feedback on the security model? The simple implementation would treat as valid any message from anyone, like with ssb-npm and git-ssb. But I sense that people want a permissions structure. If using permissions, I am thinking to do it as follows: a new message type would indicate granting or revoking permissions to a feed for a path prefix/directory. Keep using ssb-links for the index, so indicate granting or revoking a feed access by mentioning the feed with the directory name. Example:

{
  "type": "installers-grant",
  "mentions": [
    {
      "name": "https://installers.scuttlebot.io/example/",
      "link": "@ye+QM09iPcDJD6YvQYjoQc7sLF/IFhmNbEqgdzQo3lQ=.ed25519"
    }
  ]
}

Then treat mentions in installers messages only as valid if their author has permission to the path, according to the installers-grant and installers-revoke messages. There also must be some "start of authority" for anyone to have permission to anything. For that, we could hard-code a set of grants, or allow for subjectivity by considering the current sbot id to have permission to the root level (all domain names). Possibly read additional permissions from the sbot config.

Variations on the idea:

  • Have a fixed directory structure for permissions, like <domain>/<group>/<project>/<version>/
  • Drop the domain name and assume a single domain name for all paths. For the mention name, use a fixed prefix before the path, like "installers:"
  • Have a simpler model where there is just one set of authors with permissions, no subdirectory permissions.
  • Have a more general message type than "installers", if this is just a general-purpose permissioned file system.
  • Represent directories with messages. Instead of specifying full URLs, specify the path in the current directory and reference the message representing the directory. This would allow for renaming or aliasing of directories. It also might help with subjectivity, by being based more on cypherlinks than a global URL space. It would allow for not depending on ssb-links if there is some initial mapping of domain names (or root namespace) to message ids for directories, since all other queries could be done via message backlinks.
  • Rather than making new filesystem namespace thing, use/integrate an existing one like ssb-web, saljut, ssbdrv, git-ssb, or ssb-npm.
  • Rather than a permission grant/revoke model, use a membership greet/shun model, as in Liquid Association / Completely Subjective Groups.

Other concerns:

  • Would this be polluting the mentions namespace to use URLs to name blobs and feeds in mentions? The blob names might show up in Patchbay's blob summoner. Feed mention names I'm not sure if anything is using.
  • There may be issues around state resulting from permission revocation. If someone's permission is revoked, should all their existing edits be considered invalid? Should a revocation mention their last "okay" message, so that future people can tell which of their edits were done before or after the author lost permission? (What if the author was granted permission by multiple authors and they revoked it at different times?) Should edits that have become invalid due to recovation be deleted from servers? Simple solution is don't delete anything in cases of revocation, and in cases of bad content, users can publish overwriting edits. This assumes source timestamps are reliable, for new people to compute the file system state. Simpler solution is don't have permission revocation. Even simpler solution is don't have permissions, just relying on the contact graph / ssb feed replication.
  • Is permission management by grant/revocation messages the best approach? Other ideas: Liquid Association, Completely Subjective Groups, capability keys, reuse the social graph, use a new social graph.
  • Is subjectivity important? In practice, for the current motivating use case (installer shell scripts on a web server), the system just needs to run on a single server. (It could be on more, even for the same domain name, but I don't know if there is any interest in that.)

I would like to find and implement a "simple as possible, but no simpler" solution. I also want to build on familiar concepts and ways of working - but a lot of those are centralized.

Related: Trusting the SSBC

cc @mixmix @elavoie

@mix %YB582+e9IrzTW8iHiof9TlZ3nZ1ck+RWcCvPsL8aFnY=.sha256

cc @staltz @arj @kira @christianbundy @cryptix @cft @aljoscha

who else might have input on this

User has chosen not to be hosted publicly
@Christian Bundy %PNGRnwEeECilHNpblggaJfZ1/2GOZwQMQg5vOUpGE8c=.sha256

@cel @mix

I'd love to give feedback on this, but I don't think I have any of the context on what problem this is meant to solve. If possible, I'd love a way-zoomed-out intro to the problem, maybe which solutions we've tried so far, and some background on the solution.

If we're just talking about translating URLs to blobs, my first inclination would be:

I suspect I'm not grokking the problem we're solving? Hope this was helpful to explain my understanding so far.

cc: @RangerMauve, this reminds me of your latest blog post and you may have some great feedback

@cel %+Ae2g7S2bfTFBOvglXE7lhp+PVA0ENRm/1bgW6+wLww=.sha256

@Christian Bundy The problem as I am currently looking at it is how to decide what installers.scuttlebot.io serves.

I want the scuttlebot.io domain to be dedicated to the SSB network commons. git.scuttlebot.io hosts git-ssb-web, and viewer.scuttlebot.io hosts ssb-viewer. This thread follows the idea of a installers.scuttlebot.io for hosting installer scripts, for the use case of installing git-ssb, since making git-ssb easy to install was identified as something I should work on.

In git-ssb, I felt pushback for the access model that "anyone can push to a repo". So I am wary of implementing another system where any message published is considered a valid update. The problem then is how to decide what updates to consider valid, and how does that change. Updates as I am viewing it would be messages declaring changes to what installers.scuttlebot.io is hosting/serving. How does the network decide what this (sub)domain serves? (Can this be generalized to other domain names?)

@Christian Bundy %KfYJ/b/ySNawIxtRDQKjF2RLu7zFidlydtj28jSvqYs=.sha256

@cel

Thanks for the explanation! Reading between the lines, am I correct to assume that the main problem is that we want the same install.sh file to point to different blobs over time?

This problem space feels like it has a big overlap with the #ssb-wiki problem, maybe there's one solution that would work for both? For example:

{
  "type": "wiki",
  "path": "install.sh",
  "link": "&qNy5xZ4bhu61z4k+XenbtDnt1IJp0b/gLJ/vdzd5vxM=.sha256"
}

If you had a button to give my feed a subdomain, maybe that blob would be served at either:

How does the network decide what this (sub)domain serves? (Can this be generalized to other domain names?)

The simplest way would to make one subdomain per public key, like how Dat does it. I'd love to have a big distributed wiki where my content is distributed to a bunch of different SSB-related domains.

@Christian Bundy %KcIX6oJGWsy3qZtlT2aJ6WRONJdFyZFOKR2THT6ry5I=.sha256

@cel

...or maybe this would be a better task for git-ssb? I think the biggest win at the smallest cost is giving everyone a feed-specific branch that only they can write to.

@cel %1ioFH4S7CBuPeJqHVu0EZdYW2ahf3fXojwEg6zmRmS8=.sha256

@Christian Bundy thank you for the feedback.

the main problem is that we want the same install.sh file to point to different blobs over time?

Not necessarily. The URL contents could be immutable. Or could be non-editable but deletable (like npm package releases). The example in %9c2tLY9... includes a version number in the URL. But we might want mutability anyway, since it makes the system more useful and in some models would not add any complexity.

Yes, it could be similar to a wiki. But I think of a wiki as having pages that multiple people can edit. A feed per domain means each page (URL) can be edited by only one author. Also, I expect wiki to be published in a markup format and rendered on the server before serving as HTML. Shell scripts would not need that.

The simplest way would to make one subdomain per public key, like how Dat does it.

Noted. This wouldn't work for the http://installers.scuttlebot.io/… idea if just one feed has access to publish to it. Unless that feed can grant access to others. That gets to what I am wondering about how to make work and what is needed for it.

I'd love to have a big distributed wiki where my content is distributed to a bunch of different SSB-related domains.

What would the different domains be? The same system hosted on different pubs? Or different applications/variations/structures?

maybe this would be a better task for git-ssb?

That is an option.

@Christian Bundy %rUIW1jUW2eBGny+NnxVPrF0mv6f6gGrZ1GmkFUNE494=.sha256

@cel

maybe this would be a better task for git-ssb?

I think I should be more explicit. Git is a great system for managing the state of a filesystem with an append-only log. Git-SSB is a great system for using Git via SSB, the only problem is that we don't have any protected branches.

We now want to manage a filesystem via SSB with protected branches, and are discussing different ways to do that, but I'm wondering whether this might be The Perfect Opportunity to add protected branches to git-ssb and solve both use-cases.

Even a simple policy like "commits to branches that start with git-ssb-protected-* will be ignored unless you're pushing to git-ssb-protected-<your-feed-id>" would work fine, although it's a bit verbose. So if you checkout git-ssb-protected-@alice.ed25519 you can be 100% sure that it was pushed by @alice.ed25519.

What would the different domains be? The same system hosted on different pubs? Or different applications/variations/structures?

Yeah, I was thinking the same system on different servers, although you could have a handful of different software systems implementing the same behavior.

Unless that feed can grant access to others.

I think we're going to have to resolve the multi-writer problem in SSB as a whole before we'll be able to solve multi-writer in subsystems. I'd love to be wrong, but I think that any solution we could implement within the scope of a message type would probably work just fine in general and maybe should just be deployed to SSB.

@mix %35lvayG1uBkXuG5vM88avZvAzTuWQ8GnhBC7vfuvdog=.sha256

Ok I've read your message @cel but don't have time to read all the replies here in detail. Here's my take :

  1. don't boil the ocean, start simple
    • just hard-code feedIds in a config file on the server which you trust
    • this is good because it requires less new messages, and we can also assume that if a person has access to the (pub) server, then they have the right permission levels
    • we can build some complex permission system later if we need to
  2. use tangles
    • so, you and I are both maintaining some install.sh, I want to cut a release at https://installers.scuttlebot.io/git-ssb/install.sh ... how do I do that without colliding? which one wins?
    • so cel starts a new tangle thread with an initial state, then I come in and mutate it. The server decides whether my mutation is valid based on whether I'm in the config file. Actually we can also do access-control as part of the tangle ... I've been working on this with ssb-tangle
    • if we use tangles we can think in terms of a "document" which we can mutate, it also gives us a very clear history
    • fields like { url: link, blob: BlobId, tombstone: { date: UnixTime } }
    • we'd still need a protocol about what to do if there are 2 tangles trying to bind to the same url ...

I notice you're thinking of solutions which fit into ssb-links well, but I guess I feel like that's just using a low level tool and trying to fit the problem into it's constraints. (I'm likely doing the same with tangles!)

  1. Pub announce new releases
    • I think the server (pub) should publish a response to setting up serving of the latest blob. I think it would actually be really good / important for it to post a ":+1: published" message to acknowledge and announce new state
    • this could be in each a installer tangle, or maybe just a general message, or maybe in an RSS-like tangle which is just announces of all changes it's made to it's state ... this would be nice because if the server needed to be rebuilt it could "replay" itself into the correct state
@Anders %y/uXoRDFU1nOWylss0+1xSFT0UAklRGJbfOzNwGuLxE=.sha256

I really like the approach that @christianbundy outlines with 1 subdomain per feed. That leaves the problem of what the "official" git-ssb install.sh should be.

I'm really keen on some sort of system where blobs are verified by other users on the system. And once a certain threshold is reached, the server will switch to the new version. This has multiple benefits. One is that its much harder to attack. Secondly the verified could mean that they ran some tests or something else, in this case it would be a way to go from rc to final.

I have been thinking of a similar idea for blobs that contain the latest state of a number of feeds for the browser system I'm working on.

Join Scuttlebutt now