Diary of a Fork (& How to Recover from it)

source:www

What is this post about?

You may have heard of the perils of forking your feed. This post covers what a fork is in this context, how I diagnosed my fork and then how I fixed my fork

What is a fork?

A fork is a spikey utensil which you stab items to put into your mouth.

A fork can also be a life choice to make where you have to select between different select options with different possibilities and options and trials and opportunities. The word "fork" has been used to mean "to divide in branches, go separate ways" as early as the 14th century (according to wikipedia)

Here's what wikipedia says about it in a software case:

In software engineering, a project fork happens when developers take a copy of source code from one software package and start independent development on it, creating a distinct and separate piece of software. The term often implies not merely a development branch, but also a split in the developer community, a form of schism.[1]

Free and open-source software is that which, by definition, may be forked from the original development team without prior permission, without violating copyright law. However, licensed forks of proprietary software (e.g. Unix) also happen.

this picture is of my workmate jaromil and is a reference to the devaun.org fork of debian

What is a fork in the context of SSB? (spolier :: it's a Bad Thing (tm) )

This is what happened to me. It means that your feed creates a branch which a subset of your friends can see. This subset can be 0 or more. In my case it meant that 0 people other than me could see the posts I was making.

How does a fork happen in SSB?

There are a few different ways:

• you can try running same identity (private key / secret) from multiple devices. When you post from both devices but one or the other of the logs have not caught up with eachother you can then create two distinct forks.

• it used to be the case that if you changed your local machine time and made posts "in the future" then this could cause a fork. @elavoie talks about that more here.

• other

How do you Diagnose a fork?

In my case I am working with the #mmt crew and I linked a post I had made out of band (i.e. shared via www rather than inband - within SSB) here is the post: %8//8qB/...

@peg said: I can't see that.

Sometimes gossip replication can be slow, so it could just be a question of waiting some. but this got my spidey senses tingling.

From that post onwards I noticed that no-one had responded to or liked any of the posts I had made in a 24 hour period. Some of these posts were direct responses to active discussions. This indicated to me that perhaps people weren't seeing the posts.

I then asked some of the #mmt crew to make a post with a link to the last post they could see in my feed.

@mix did that here: %cQDfDfG...

hey @dan hassan here's the most recent message I can see of yours : %4Ax2Ikb...

I also checked from my manyVerse account and could not see any posts.

Recovery

Running out of time, so I'll keep this brief.

First I did diagnosis.

I captured sbot state by running sbot progress to see what the general index number was. This is not perfect as it is subjective. There is no objective target to measure completeness. The aim was to recreate the DB from fresh.

DO NOT interact with the fresh install. Don't like anything. Don't make any posts. Do not add a name about yourself.

• I moved .ssb -> .ssb-backup

• I deleted patchbay && patchwork

• I created a new .ssb directory

• I copied .ssb-backup/secret & .ssb-backup/gossip.json -> .ssb

• I cloned patchwork and installed and ran

• I left patchwork running

• I noticed on watch sbot progress that syncing seemed to stall

• I closed patchwork

• I cloned patchbay

• I ran patcbay

• I noticed watch sbot progress now included ebt data and syncing resumed and finished.

• I then made a post and checked from my #manyVerse account to see if I could now see the posts. I could. Yay.

Final Thoughts

This is just the way that worked for me this time. I am not saying this is a canonical approach that will work for anyone else.

Diagnosis

I think I made an attempt to post a reply to a topic to which the root was from a blocked user or someone outside my hop range. I noticed that my App hung and that the post I made was not in my log. It is from that point onwards that my feed was forked.

I think

tags: #bugs #ssb-bugs

Links to other narrow fork escapes:

@kieran' grand adventures

@rabble writes:

Mine was caused when i was connecting over my vsat which is very high latency and often drops packets. I fixed it by deleting everything but the identity... and then also reusing the gossip file from an alternate ssb account.

@elavoie writes:

In my case, my clock was drifting, I posted from the future, then changed my clock, and ended up posting a message with a timestamp that was earlier than the previous post. That violated the monotonically increasing timestamp assumption in scuttlebot. Dominic fixed it by relaxing the assumption for posts in a given time window.

I am not quite sure what is the state right now but I believe the monotonically increasing timestamp assumption has been removed so that should not be causing your issue if people use a recent client.

@christianbundy if you have the commands you ref here:

<Christian Bundy> were you able to repost all of your previous posts?

<Christian Bundy> in the past I've taken the output of sbot createHistoryStream when I'm forked and then passing it to sbot publish after some jq magic

I would find them super useful!

@dan

I think it was something like this:

seq="1337"
ssb_id="$(sbot whoami | jq -r '.id')"
stream=$(sbot createHistoryStream --id $ssb_id | jq -c "select(.value.sequence > $seq) | .value.content")
for msg in $stream; do sbot publish . <<< $msg; done

In this case, seq should be set to the last message that successfully replicated, which is a point before the fork.

At the time I had an issue where newlines were getting garbled before sbot publish ., but I think that's fixed in the above version. Please let me know if you use it and whether it gives you any trouble! For safety, it might be wise to add a prompt to sbot publish since this solution is so experimental.

@happy0 gives an excellent related answer here:

I'd strongly recommend you don't share your key between multiple devices to use your same scuttlebutt profile on them. You might run into this situation:

1. You have posted 10 things on your profile. Device 1 is offline.
2. You post something on device 2 (you now have 11th post.)
3. You post something on device 1 before it has synced your own feed from other computers.

4. You now have 2 versions of your 11th post. Other computers have 2 versions of your 11th post.

5. Any time you make some posts from device 1 or 2, some computers reject them as invalid as they have a different version of post 11 (and all the posts onwards.)

6. One of your feeds is now invisible to a lot of people after post 11 as they agree that the 'real' post 11 is the other one.

To be safe, I'd use 2 different keys and wait for the 'sameAs' functionality to be implemented that allows you to safely share an identity between devices.

If you must share the same key between devices, then it should be safe if you share the entire .ssb folder (the database, not just the secret file) over a network share between the devices. That way all the devices always have the same view of the database as they post as it's the same underlying folder.

Could there be a way of telling people who'd followed the fork that they needed to delete the feed and re-request it?

@rabble if your feed forks and both forks sync to other peers, thus creating a network partition of your feed, your feed is effectively dead.

the protocol by design does not have a concept for two valid messages with the same sequence number. the simplicity of this design allows us to avoid entire schools of computer science complexity, i think rather than try to recover dead feeds we should try to improve the user experience of creating new feeds (as an returning uxer), as we will also need these experiences for key loss, key revocation, and key rotations.

Seems related to empowered covens

#EOF 's and #tombstone 's

"fork proof": %EH8yUI1..., %iFy5K3Z..., %1JVLpZd...

Total newbie here, and reading the above leaves me confused. I've been using patchwork less than 24 hours. My second or third post was marked as a #fork by patchwork (in some status line attached to that post) which at the time I assumed meant that I successfully started a new sub-conversation.

Now, I'm not so sure. Why would patchwork tell me I have a fork? If this is bad, shouldn't it tell me "that's bad", and suggest how to resolve it? Or auto-resolve it? After using patchwork for maybe an hour, I need to already worry about arcana?

It's just a channel named #fork, @Linas, don't worry.

@christianbundy's fork recovery approach from developing right at the bleeding bleeding anemic end of the stack

Total newbie here, and reading the above leaves me confused. I've been using patchwork less than 24 hours. My second or third post was marked as a #fork by patchwork (in some status line attached to that post) which at the time I assumed meant that I successfully started a new sub-conversation.

Now, I'm not so sure. Why would patchwork tell me I have a fork? If this is bad, shouldn't it tell me "that's bad", and suggest how to resolve it? Or auto-resolve it? After using patchwork for maybe an hour, I need to already worry about arcana?

Hey @Linas thanks for the question!

This is a language clash :-D Latest patchwork has a new design implemented. If you click the reply button to a specific post, then patchwork interprets this as "I want to create a new subconversation thread which is directly responding to this particular post". If your comment is a reply to the overall thread and conversation as started by the first opening post, then the design of the interface wants you to write your reply in the bottom reply box.

So, what you have surfaced is a semantic confusion are forking. This is for sure influenced by the early participants of this system being quite technical and so using language such as, please fork the thread, which actually is not a universal concept....

so, for example, this reply is "off-topic" in relation to the opening post of this conversation and so in other cases we might chose to "fork" the conversation to make a new topic specifically to do with this... so... lets do that and we can discuss better semantics there!