ssb pack/unpack

We've spoken quite a couple of times about SSB replication over USB, for sneakernets. In other words, it would require packing a local SSB database into one file in the origin computer, then in the destination computer, the file could be unpacked/consumed to update that local database with new data, if there are any.

Technically, it would require:

• A file format and/or folder convention
  • I'm interested in it being like java JAR or Android APK: it's just a zip file, but with a custom extension
  • Example: staltz20190319.sbp but internally it's a zip file
  • I chose .sbp as an initial suggestion. "Scuttlebutt Package". Web searches tell me that sbp is used only by a few other not-so-popular applications
• A JS library and CLI to pack a local SSB database into a .sbp file
• A JS library and CLI to consume an .sbp file and update your local SSB database

Here's the fun part: the diversity of use cases for this are huge! Of course the canonical use case is sharing the package through USB, but there's more:

• Share the file through email
• Share it through Firefox Send
• Share it via Bluetooth
• Share it on a website as a download

Then SSB apps can register themselves in the OS to be the preferred app for the .sbp file extension, and this would make it even easier to consume the file without doing manual work. Android as an OS also supports preferred apps for certain file extensions.

@andrestaltz, do you remember BBS? the dialup kind running PC Board or Remote Access? If you were using it back then, there was a very popular solution for message boards and offline viewing/replying to threads called qwk files) and also Blue Wave files) which were compressed files carrying the messages and threads. It is basically what you're trying to build here and for me still the best UX ever. I miss Blue Wave reader a lot. Here are some screenshots of MultiMail which was similar to Blue Wave (I couldn't find screenshots for blue wave).

Anyhow, if the file format is easy enough to write and read, this could sidestep muxrpc and flume and be a new way to SSB with friends.

@moid: I'm imagining using SSB feeds in healthcare apps as part of medical records

Great use case

@SoapDog: do you remember BBS?

I never used it but it does sound quite similar to what we could build here :)

@keks: currently working on extending the friend graph to properly support feed access control

Hmm, sounds tricky indeed. Something to ponder about until we meet in DTN

@keks @andrestalz, could a first pass prototype be made using the following characteristics:

• pack your feed into files + control files in zip format.
• only include feeds which have been cleared for ssb-viewer display as those people have already vouched that they are ok with being publically viewed.
• work one way only, you just view it (or import it).

This might be not that hard to build and show the potential for helping with initial sync.

@cft, I love TUIs, I grew up with Turbo Pascal and BBS. I am an ANSI person. Wish that kind of software was still easy to build. Heard good things about a Turbo Vision alternative in the fpc community.

Thanks for the protocol spec. I will check it out. I've never seen that one before.

@SoapDog: only include feeds which have been cleared for ssb-viewer display as those people have already vouched that they are ok with being publically viewed.

If we do like that, then it would be impossible for Alice to get Bob's feed through the zip file, if Bob has chosen publicHosting=off, even if Alice is friends with Bob. So I think we need to skip the publicHosting verification.

I would actually simplify even further by packing a specific feed for each zip. This way we explicitly acknowledge who's data are we exporting, but also it's an easy way of starting to write this code.

oh, I was not trying to solve the problem, just thinking an easy way to build a proof-of-concept. Maybe just packing each feed as its own file, and selecting friends only, and then zipping them all in a single file might be better right?

There's a exporter/inporter that @arj wrote:

https://github.com/arj03/ssb-exporter

On #scuttlecamp we had an awesome chat with @noffle about mapeo. One of the subjects was porting it to use SSB, and I have great interest on that for the work I'll start with indigenous communities around the Amazon. As Mapeo's initial use case was to carry data around using USB, that would have to be a must to have a SSB version of it.

i love this thread. remote communities which I am peripherally connected to would find this immensely... not just useful, but a pre-requisite I think...

@andré

Have you already seen ssb-exporter? I think this may do what you're looking for, but I'm not completely sure.

# export all posts from your database into a simple JSON file
./export-data.js -e feeds.json

# import posts from a simple JSON file into your database
./import-data.js feeds.json

Oops, it looks like that should be feeds rather than feeds.json -- I think it exports a directory, not a file. My only experience with this has been my one pull request.

@cinnamon "@noffle and I were talking about SSB-over-sneakernet using an SSB drive. I think we were planning to make the USB drive into essentially a portable pub with a standalone SSB server application on it. I wanted to call it a "mixtape", like a collection of the latest content that you would give to a friend."

<3 mixtape as a name and concept

^ found this in the logs looking for "tape"