Introducing... SSB Mirrors!

Simply put, it's a pub that replicates just you

Example: http://ssb.staltz.com a.k.a. @mirror_andrestaltz

Keep this only on SSB before we announce it ready to the world, next week

Read the readme here: https://github.com/staltz/ssb-mirror/tree/readme and try setting up your own.

I've been working on this the past 4 days, and now it's finally ready. Mirrors are alternatives to Pubs, where the sbot will replicate only one SSB account. It's a simply idea, but it has some interesting benefits:

• Quicker sync if people are interested in getting only your data
  • It took about 4min for an empty Manyverse app to sync with all of @QlCTpvY..., and that's an old and heavy feed
  • If you mostly want to broadcast content to the world, RSS style
  • Quick initial sync is very important for onboarding, people can experiment with SSB much more easily like this
• Open invites makes it easy to onboard people onto SSB
  • Also important for newcomers who want to casually try out SSB
• No concerns about hosting other people's data
  • Removes the legal concern, and the randos lurking in pubs
• Lightweight server to run and maintain, because it holds only your data
  • I think the storage will be probably at most 5 MB (log) + blobs (I don't know how much blobs would require, depends on the feed)
  • These are light requirements also on bandwidth

The downsides are:

• It is unidirectional (people get your data but you won't get theirs, unless you follow their mirror or have a pub in common)
• Holes: New users onboarding SSB with just your data will still have to reach out for more data from other community members, to have a meaningful interaction

The other big innovation here is that setting up such server can be done with zero DevOps skills, you just need to know how to click on websites. The trick is that the first user who accesses the website gets to command the server to replicate them. This first user is 99% likely to be the person who just set it up. And in case it's not, you can always just create a new one from scratch.

I tried to use Heroku's Installer Button, but Heroku turned out to be the wrong platform for this. I ended up copying @mikey's ssb-pub set up on DigitalOcean for this. Thank you a lot @mikey for your Docker kung fu. I also suppose it's fine that the one button installer is re-used from butt.nz?

This is an experiment, so please don't yet talk about this openly on other platforms. Let's see if it actually works well first. I'm particularly curious what this means for git-ssb. You could have an SSB feed just for a git project, and that feed has a public mirror. You can still do pull-requests with the community, but people publicly interested in that repo wouldn't have to necessarily get data from the surrounding social graph.

Let me know what you think!

@andrestaltz Does this look like a regular pub/peer on the network? If so, once there are enough mirrors out there, there's a pretty high probability that all my gossip connection at a given time are to mirrors, which would effectively lead to isolation of my node even though I'm online. That's more of an argument against ssb's naive-yet-not-sufficiently-randomized gossip scheduler than against mirrors, but since the current gossip scheduler won't change that soon, this might be worth keeping in mind.

This is a great idea @andré. How hard would it be to change this to replicate a specific id + the people followed by that id?

@Aljoscha I was going to comment also the same problem (which, yes, would exist). And yes, it would help that the gossip scheduler knows how to prioritize "hot spots" (pubs with likely a lot of updates) simply by not sticking to mirrors. Because the invite code has SSB+Mirror embedded in it, it's possible for us to know which peer is a mirror, then we can save that in the database somehow. And yes, also good to note that the gossip scheduler won't change that soon, and there's always possibility that people are using old versions of apps, even if we change all apps.

@arj It wouldn't be hard, but that's essentially a pub with config.hops = 2, and it's quite different to host 1 account than to host 150 accounts (dunbar number).

@bobhaugen Thanks for the feedback. To make this clear, mirrors are basically pubs, they just replicate 1 account, whereas pubs usually replicate hundreds.

@Aljoscha actually, additional comment: even if we don't use mirrors, it would still make sense to change the gossip scheduler so that it disconnects as soon as (plus a few seconds) its done downloading the recent updates from a pub, and then go on to visit the next pub, and keep rotating.

wooo, this is great @andrestaltz!

agree this breaks some assumptions in the current replication implementation, but i think it's a good step in the right direction and we should improve our replication systems to be more flexible in regards to network / social graphs.

it would still make sense to change the gossip scheduler so that it disconnects as soon as (plus a few seconds) its done downloading the recent updates from a pub, and then go on to visit the next pub, and keep rotating.

but then we'd miss the benefits of long-term connections, low-latency live updates and less connection handshakes.

How hard would it be to change this to replicate a specific id + the people followed by that id?

i like this idea, which i'd describe as config.hops = 1 not 2.

@andré I was thinking something in between personal mirror and public pub. Something like a friend-pub combined with ssb-incoming-guard.

I really like this idea of a personal mirror though, a bit like a blog, it could even be hosting ssb-viewer as an option to make it easier to people to peek in.

Thanks @mikey and @arj

Having slept over it, I'm starting to realize that perhaps the best innovation of this is just the easy setup without fiddling with the terminal, and the possibility to open a webpage to configure the server.

So maybe the next iteration would be improving the page that says "Empty mirror" to be an admin page instead, and have more configuration options, and then once the user presses a submit button, it configures the pub forever (without possibility of reconfiguring). This is to avoid having an authentication layer, and assumes that the first request to the server is almost certainly to be from the owner of the server.

Really great comment, and it does resonate with a lot of ideas I'm planning for ssb-conn.

Yes to randomization.

Also yes to backup connections. I think that would be handled in ssb-conn-staging. I noticed that we might be missing an additional API in secret-stack (by the way, I know you're probably talking about overall protocol and other implementations, but I'm thinking how to implement this ASAP using the node.js stack). The problem is that we can't just use secret-stack's naive connect, because there are a bunch of hooks that trigger as soon as a connection happens, such as replication etc. There should be a way of pinging or connecting with multiserver, without triggering all those hooks. Basically we just need to create a multiserver client, but the challenge is that we need all the same multiserver plugins that are running in the secret-stack instance. This is why I think there has to be a change in secret-stack, to add a way of cloning the ms client with its plugins. cc @dominic I might PR this on GitHub.

Good comments @cft. I also had similar thoughts, but phrased in a different way. The problem with pubs is not that they host too much data (so mirrors, having less data, are not the solution), the problem is that clients request way too much data (by default) even when they don't want so much.

In fact it seems like if you're going to host an SSB server of any kind, then it's worth (in terms of efficiency) to fill up its storage as much as possible, while still staying within healthy limits (RAM and Storage limits).

This thread was very valuable to me, because initially I was quite excited to publish this to the world, but now that we know of all the implications it would have, I don't think 1 mirror per person, many thousand times, would currently be a good solution.

There are two main problems that ssb-mirror tries to solve, and I think are worth treating separately:

Resource underuse or overuse. This problem can be experienced either in servers, when they host too much data and crash frequently, or experienced in apps, when they request too much data at once and consume a lot of bandwidth and CPU power. Ideally, servers host as much as possible but not too much to be efficient hot spots for feed updates, and apps request as little as possible but not too little to be efficient consumers.

Skills required for server admininistration. Setting up servers is not for everyone, unless they invest with some learning (terminal, SSH, docker, logs, etc). Also maintaining them when they crash often is not for everyone. So one thing ssb-mirror is solving is making it easy to setup and maintain a server. I think this is the good innovation.

Then there are secondary problems, such as the legal aspects of hosting other people's data. If it's random people, it's more of a problem. If it's friends, it's less of a problem, but then you have to be careful to respect the publicHosting setting.

I think the problem worth solving with more care is the resource underuse/overuse. A server replicating 1 feed is resource underuse, and contributes to problems instead of solving them, although it has the benefits of simplicity, as a concept, and legally too.

Maybe one improvement would be a chatbot interface for the mirror to replicate more accounts, for instance:

• Upon setup, @mirror_andrestaltz sends @andrestaltz a private message asking for other accounts to be replicated
• @andrestaltz answers that private message by listing some SSB ids
• @mirror_andrestaltz includes those feeds into its list of feeds to replicate, and fetches data from @andrestaltz

Of course another way to do this is to simply configure hops setting, but for legal reasons you might want to distinguish between "I want to follow this person's updates" and "I want to re-host this person's data to the world". The latter is an opt-in commitment to rehost. We could also consider implementing this with fractional hops, e.g. friends at distance 0.9 are accounts that I am willing to re-host to the world, accounts at distance 0.1 are #sameas, accounts at >=1 distance are just normal follows without re-hosting.

Thoughts?

Very good points @gozala

@Georg

I could just setup a droplet for a friend and send them the invite code! Super cool!

Didn't realize this is also a really nice use case. Thanks!