fixing the non-monotonic timestamps

As @Fabián Heredia Montiel describes in this thread @elavoie managed to travel back in time and create a message with a timestamp that was less than a the timestamp on the previous message. I wrote a script to check for other occurances of this, and found 4 other cases, all between 19th december 2017, 7th feburary 2018.

Some people have argued for abandoning the idea of monotonic timestamps. Let me be clear, I do not like that idea, but I will not pronounce it "off the table".
However, I do insist on some things:

• The protocol should aim for stability, we try to avoid changes that cannot be unmade.
• We should avoid designs that include aspects that are difficult to change, but we can't avoid this completely. examples include: maximum message size, number of recipients, wether feeds can fork, and other message validation rules such as wether timestamps may decrease. These changes must be considered very carefully before making them.
• Sometimes, there has been a bug or otherwise messages get through that ought to have been invalid, such as, messages that excede the character length, by having utf-8 characters, because javascript's string length is the number of characters, not the number of bytes. Although these things are embarrasing, every protocol has things like this.
• eventually, once other parts are solid, we should create a (probably) binary feed encoding that fixes all those problems. And that can be the default encoding for new feeds. (we'll always need to support the old formats, if people want to read the old messages, unfortunately)
• when we realize that we messed something up (such as this timestamp thing) our response should be proportional to the scale of the problem.

Okay, those I think are loosly the principles we should follow to design a stable protocol.

On this specific issue, I want to say: monotonic timestamps are good. Firstly, requiring the timestamps to be monotonic also implies that they are unique. This is a very useful property in a database! If there are no restrictions on timestamps, it's an invitation for people to use whatever timestamp they like, in ways that affect clients in weird ways!

But at the same time, since it seems we did allow messages to disorder for a period (before ssb-validate@3.0.7) we should make that official - messages from that period have the order requirement relaxed. Okay, for this period, timetravel was briefly allowed and inparticular @elavoie used it.

I realize this makes the message format weirder and weirder, yes, this is embarrasing. But I am doing my best here. Okay various javascript quirks are sneaking in, and because of immutability we can't remove some of them. All protocols have things like this. Lets start thinking about what the ideal encoding might look like, and implement an move to that later. In the mean time, lets solidify what we have to, including those mistakes, where necessary.

@alanz that is true! but surely that is a proportional response to such an outragously wrong time, that is also quite unlikely to happen by accident. (times in the past are more likely) or times within 12 hours (especially: dualboot linux into windows, windows uses local timezone for system time, but unix (rightly!) uses UTC, and if you are in NZ that's 12 difference)

Another technique that's worth considering: if an incoming message has a future timestamp, delay it until that time. That way, local timestamps always make sense and you can't trick the system by setting your clock forward. (as in the various ways UI uses timestamps, mostly for ordering messages in the UI)

I made a PR to fix this non-monotonic timestamp problem https://github.com/ssbc/ssb-validate/pull/5

would appreciate a nod from people who will be effected by this @elavoie @arj @keks @cryptix @duncan

I wonder if this was at the root of Kieran and I's issues outlined here:

%DYS8EY7...

@corlock timestamps are only gauranteed increasing within a single feed, not between feeds. It's possible that they respond with a lower timestamp, as long as it's increasing since their previous message.

@keks and I talked some about this but couldn't really agree on a stance together with regards to go-ssb.
He needs more time to think it over and we would benefit from a face2face meeting once he is back from his trip but here are my thoughts on this:

First of all, I totally agree on the design principles but don't understand why we double down on verifying the timestamp here and I actually don’t fully understand why we need it to begin with. For replication the sequence is everything that is needed and the timestamp is just added benefit to the content, not the verification or replication.

Again: We have two monotonic counters, sequence and the timestamp.
When we discussed this ~2yrs ago, we came to the conclusion that having both is unnecessary. At least this is my take away from Removing (mandatory?) timestamps.

Additionally: until https://github.com/dominictarr/ssb-query/pull/5 everything was running on receive time AFAICT. So they weren’t even used as information even though they are included in every message.

I'm with @alanz that the clock on computers is a messy thing to deal with. NTP outages occur, misconfiguration occurs, etc. All these problems get shoved on the user.

Again2: I don't want to drop them all together but when we have a a defined, hardcoded window of exclusion for the verification.... why verify at all?

gosh this sounds grumpy.. I'm not, I'm just confused about keeping them in the verify process.

ps: the protocol guide also says nothing about this...

A week ago I was hit by a timestamp issue. I booted my computer and got online in a place with WiFi that allowed SSB but not NTP. I manually set my computer's clock (since it doesn't have a RTC). Turns out I set it wrong. I found this out after publishing a few SSB messages. I found it out when trying to log into a site with a TOTP. I set it to a more accurate time and then was going to publish an SSB message but the publish failed with Error: timestamp must be increasing. From sbot's perspective, I was trying to publish from the past. I could have set my clock back to the future and then published the message, or waited for my feed to catch up with the local time and then published the message, but did not have time to wait and did not want to use a "known incorrect" time, so instead I did not publish the message, and went on with my day. If I had accidentally set the time to a much more incorrect one, my feed could have been stuck in the future for an arbitrarily long time as @alanz cautioned in %GD42+sQ.... Maybe if peers rejected incoming future-dated messages then I could take the opportunity to fork my own feed back to the present - carefully and trustfully. Anyway, this is potentially a bad situation and indicates to me that concerns about the timestamp should be bubbled up into the UI - such as giving an indication in the composer of current times or time differences, such as from the local clock, last published message, last messages received from peers, or peers from RPC. The gossip.ping method can be used to ask peers for their time. That method was actually what I used to set my clock manually; next time maybe I will make a script to do it correctly.

Or, don't require peer's messages to have monotonically increasing timestamps.
If people engage in time travel, whether to recover from mistakes, or for other reasons, it is okay with me.

I see usefulness of socially-validated timestamps. For example, ranking algorithms like on reddit and HN use timestamps of submissions. But I'm not sure if this important enough to enforce as a network requirement in the replication layer. I would lean towards not requiring what is not necessary. But adding the time window exception for @elavoie would be better than the current behavior.

Thanks for sharing your story, @cel! This is exactly what I had in mind... imaging how to explain this to some1 less knowledgeable gives me support-shivers, though...

Sounds like a early 2000 game cracking tutorial.. “yeah, just set your time to Y, post this and then set your time back to now, post again and then it should work again as usual”

I see that we need to improve this I just don’t get why the whitelisting range is better than stopping to check this altogether.

Okay, I'm coming around to the idea of removinging mandatory increasing timestamps. monotonic timestamps could be a should not a must. I realized that I could still do my social proof idea with sometimes going back timestamps, because you can still measure wether a particular feed does have sensible timestamps.

I think removing timestamps all together, is a no-go though, because they are too useful in the app layer. If you want those removed you'll have to convince @matt and @mix

monotonic timestamps could be a should not a must.

And while I could envison posts that don't need timestamps I'm perfectly fine with keeping them to keep stuff working as it is as long as we don't verify them.

They're also fine for human consumption (“50 minutes ago”, etc.), aren't they?

Which makes them part of the application layer, not the protocol layer. If your client wants to display those messages, it can look for a timestamp key in the message content.

Since we are on the topic of timestamps: What is their data type? Unsigned 32 bit integer unix time? Unsigned 64 bit integer unix time? An IEEE float (32 or 64 bit)? A series of characters matching the regex [0..9]*? Or [0..9]+? Or perhaps [0..9]*(\.[0..9]*)?? What about NaN, Infinity?

When I got a floating point timestamp when testing the rust implementation, I literally stopped working on it for a few days, out of frustration...

monotonic timestamps could be a should not a must.

@Aljoscha the timestamps come from https://github.com/dominictarr/monotonic-timestamp.

Unless someone wants to suggest a way to make an index which causally sorts to whole DB, removing published-at timestamps is not an option. Restoring your DB, or syncing for the first time would be a nightmare otherwise

Re: format, Piet and I have started using ISOStrings for timestamps in scuttle-poll :

> new Date().toISOString()
'2018-06-09T02:03:18.707Z'

What we've noticed so far:

• a human can read them
• they can still be queried against because ISO time stamps are ordered Large >> Small !
  • bytewise / charwise comparison which the flumeview level uses works

@Aljoscha the timestamps come from https://github.com/dominictarr/monotonic-timestamp.

@dinosaur The problem is this: I don't care at all how @Dominic computes his timestamps. Ssb is a distributed system, anyone can throw anything at you. In fact, all users who replicate @Vendan have timestamps in their database that were not generated by the code you linked to. There simply is no place where "timestamps come from".

So when I ask for their type, I really want to know which kind of data has to be rejected, and which data has to be processed, as mandated by the non-documented protocol. Here are some fun cases that come to mind:

0
42
001
0.0
1.0
0.1
0x1B
0x1K
-1
-1.1
1.2e-37
1.2E4
1.2E+4
""
'single-quotes'
"cookies?"
":clock4:"
"2018-06-09T02:03:18.707Z"
undefined
null
NaN
Infinity
+Infinity
-Infinity
[]
[0, 1]
[0,]
{}
{"a": 0}
9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999

And then there is the question of how valid timestamps should be treated. Parse into a float (how many bytes?)? Parse into integer? How do comparisions work? As floats, or lexicographically? How do I compare 01 and 1?

Yet another question: How should invalid timestamps be treated? Ignore the message? Block the complete feed?

This is the stuff that matters. I'm not asking these questions because I want to know the answers (honestly, I'm afraid of them), I'm just stating them, because they matter more than any piece of code you could link to (or I could search for).

And I forgot a whole class of problems that come with the limited precision of floating points. There's a few ambiguous cases you can construct. And good look with the (or should I say "a"?) json spec not clearing those up at all. There's a bunch of implementation-specific behavior hidden in here, and it is already baked into the protocol.

I'm sorry if I sound bitter, but these are real problems with the protocol. "We'll use json, that's easier" and "We'll add timestamps, that's easier" add up to a complex mess, where not a single person, not even dominic, knows how exactly everything works. These are the reasons I have become rather inactive, and I already feel frustrated just from my last few posts.

As much as I love the design space ssb is exploring, the protocol details prevent me from committing to it. I don't even want to blame anyone. All design decisions are trade-offs, and the ssb protocol as it stands allows rapid development, high experimentations, and fast exploration of the design space. But these come at the cost of the ability to stabilize the protocol, and of long-term maintainability. I joined the community in a phase where the importance of these properties already shifted to the latter. So even if intellectually I can understand/retrace/justify the protocol design (or protocol evolution?), I mostly feel frustration.

Huh, this post turned out different than I expected... But there have been (and are) quite a few other technically-inclined users with similiar feelings, so maybe this should be discussed more. CCing a few channels/humans who might care about this meta-protocol stuff. Feel free to add more, I'm lacking the energy to go on a search right now. #butt-studies #community-gardening @corlock @Teq (you were part of the research project about people's motivation on the network, right?)

appreciate you sharing your frustration so openly @Aljoscha - it's really important for us to hear about what's blocking people, especially when it's this demoralising.

I personally think we're still at a stage where it's possible we can (socially) afford to reset the ecosystem if there is some impassable problem which requires foundational changes. As in there's a lot of content here, but there aren't really thousand of people yet.

I might be reading this wrong buuuut, can someone clarify a case for me which should affect me soon: If I change timezones and post a message which is timed in the past from my original timezone point of view, am I in trouble?

@SoapDog changing timezone doesn't change the timestamp, it uses Date.now() which is always in UTC.

I'm almost certain this comment won't be met with much support (so I apologize in advance) but I've been thinking a lot about decentralized clock synchronization while helping to write the FOAM whitepaper. It's probably well known within the #scuttleverse that ssb implements a completely subjective view of the network, down to the ordering of events, which I think is one of ssb's most interesting features. Proof of work blockchains choose the exact opposite approach, complete state replication and an objective, totally ordered clock. I'm wondering if ssb might want to make this choice explicit by allowing users to chose which clocks they trust. This could be the personal clocks of other users, or an external proof of work clock. The latter which would be accomplished by inserting the the most recent block hash into ssb's message header. Each user would then have the ability to set the precedence of different clocks. I think this is more of a thought experiment than an actual suggestion, but I like the idea of letting users choose how they relate to time.

After consulting with dominic, I've published ssb-validate 3.0.10 that removes the monotonic timestamp requirement.

@elavoie I think the question is rather what needs to be updated and who can do it

things from top of my mind :

• ssb-validate > secure-scuttlebutt > scuttlebot

  • all the clients

  • all the pubs

    • ssb-easy-pub
    • mikey's docket pub setup

If the pubs aren't updated with this people like elavoie will only be able to do LAN gossip!

@arj does that sound right? If it is I think it might be good to launch a thread to notify everyone and check off all the things. hmmm this is kinda gonna force people running older scuttlebots to update ... which is good I guess, because it will help pull ebt into effect more?

Correct @mix

I have been running similar code on my pub and client for a while, so messages will get through soon enough if your client is updated.

I have pushed an updated package-lock.json so that patchbay gets the latest version. For patchwork @matt might include this minor update in the final release of 3.0.10.

you edited the package-lock manually?
Isn't it more like we need to bump the version in secure-scuttlebutt, then bump that version, then bump the versin of secure-scuttlebutt that scutlebot uses?
I get that a patch version should propogate but I think we want to be more proactive with this one ?

FYI looks like running npm update isn't enough to bring in ssb-validate@3.0.10.

From https://docs.npmjs.com/cli/update:

As of npm@2.6.1, the npm update will only inspect top-level packages. Prior versions of npm would also recursively inspect all dependencies. To get the old behavior, use npm --depth 9999 update.

I ran npm update --depth 5 and now the version has been updated correctly

@matt is there anything in the way of cutting a new patchwork release with this change in place? there are still people who can't see @elavoie

@Dominic

I need to fix a bug in private message notification pretty urgently, so I'll fix that and then push out a new release!

Okay, new release is published!

@fabianhjr I have not seen you in the longest time! So it seems I have not been seeing you!