Another question for the client devs: Would you be okay with timestamps becoming optional?

Optional timestamps are a compromise where we still keep timestamps in the protocol, but a uxer can explicitly say "I don't want to supply a timestamp for this message". There are valid reasons for that decision. An explicit opt-out can be handled directly by clients, whereas the current anonymization mechanism of timestamp randomization causes clients to behave weirdly. Given a message that does not have a timestamp, here are a few things a client could do:

• ignore it
• sort by receive-time
• sort by causal order, break ties (i.e. incomparable messages) by treating nonexistant timestamps as negative infinity, or by receive time, or by whatever else makes sense
• diplay a large, red banner with the text "User @foo chose to anonymize their timestamp. We don't want their kind here. Please block them.", i.e. an explicit version of what patchwork currently does implicitly (Sorry @Matt McKegg, I just can't help myself. I sincerely hope kiwi culture doesn't consider these snarks as too rude. If these are not ok, please tell me, and I'll try to adapt my communication style.)

The ssb-client API could simply omit the timestamp entry of messages which chose not to give one. Or if you prefer for things to break more silently, it can include a timestamp field, but set it to NaN.

Do note that a malicious user could still supply insincere timestamps. Timestamps being inherently broken is not a thing that stopped being true. But with optional timestamps, these would clearly indicate malicious behaviour. For which ssb already has a mechanism, blocking.

CC @mix @Matt McKegg @cel @SoapDog @Tim Schumacher @christianbundy @ev @arj @happy0

@Aljoscha

Would you be okay with timestamps becoming optional?

Sure. I think that for a message to be a valid type: "post" though, it should contain a timestamp. It would be up to the client to enforce this (maybe a pre-validation before rendering like patchbay does).

I can imagine that there are many uses of ssb messages that don't require, or not make sense, to have a timestamp. And since the timestamp is almost entirely used for UI stuff (not protocol, replication, etc), I think it makes sense to be at the message type schema level.

I sincerely hope kiwi culture doesn't consider these snarks as too rude.

I reckon patchwork should automatically block feeds that don't include timestamps! haha! No, not at all.

mmmmm.... not sure. I think matt is right about post type really benefiting from timestamp. It's really annoying reading things that are old when you want new things and a person just came into gossip range of you. or rather, it's strange, and sometimes embarassing when you don't realise how out of time your reply might be.
It's annoying when a whole persons feed sits on top of all your other friends content, that makes me tend towards blocking to remove the noise.

I think if you remove timestamp you might have to make branch compulsory.
I think branch is actually way more interesting that timestamp.
Downsides would be losing ability to do any easy queries.

Of yeah I tihnk as the database gets older being able to say "only message published in the last year" might become really useful. I think this might be the case with all message types. Something like tags don't really need a timestamp to operate, but at the same time, if I want to see new tags my friends have been gathering how do I do that.

I think we need to imagine a range of scenarios and propose how to build those quiries. Or to spend a week re-writing modules and avoiding using timestamps to see how hard it is.
I am against anything that makes querying any harder than it currently is - either I don't have good strategies or some queries are really hard already.

cc @Aljoscha @matt

I guess optional timestamps essentially boil down to the following: Client devs can no longer ignore the fact that there are problems with timestamps. They are free to choose to ignore these problems (i.e. by requiring them and ignoring unstamped messages), but it is no longer the default choice.

Which would prevent situations like @SoapDog's "I have no idea what happens". I'd consider that a vast improvement.

I think missing or randomized timestamps would be fine. I imagine in many such cases, backlinks (replies, digs) would offer some meaningful idea of the time a message was published/replicated - as would, in some cases, looking at previous or subsequent messages in the author's feed.

patchfoo sorts posts by received time on /public, /private, /mentions, /live, and channels. (On all except /live, this can be changed to sort by claimed timestamp using ?sort=claimed). In a message thread page, ssb-sort sorts messages by causal order + timestamps.

@Aljoscha I agree with @matt here -- it'd make the most sense for timestamp-less messages to be a different message type than type: "post".

I'm having trouble imagining how these timestamp-less messages will be used, would you be up for describing your application idea?

I'm having trouble imagining how these timestamp-less messages will be used, would you be up for describing your application idea?

@ev ssb at its core is a database, not a social network. Think of an arbitrary database on this planet that contains entries without timestamps. You just found a use case for timestamp-less messages.

hey @Aljoscha did you read what I said?

Luckily I can see that your branchrefers to my message which means I don't need to worry if you received what I wrote, I just need to wonder whether you read it / heard anything I said

(bold feels a bit like yelling, which feels a little like listening)

I think what you smelt @bobhaugen was me trying to signal I felt the comment was polarising, and I didn't enjoy it. I could totally have misread it.

I felt like I posted "I'm totally up for hearing this problem, and I'm interested in exploring the broader dynamics together, here's how I think we could do that".
And what came back was "clientside cannot ignore this"

I read it like that because I was signalling I was onside with exploring this and there wasn't an acknowledgement of that. If that didn't mean anything then perhaps they were doubling down with a pushy assertion. Readying again they could have just been musing and re-phrasing. Anyway thanks for raising that @bobhaugen and apologies if I read wrongly. If there was something else going on for you @Aljoscha that needs sayin, I'd love to hear it.

I don't assume @Aljoscha is stupid at all with regard to knowing what's what with how low level ssb works. I think they're far more experienced.
I think I'm more experienced in building clientside, and I think by working together we can figure out some awesome solutions.

Sorry @mix for not giving an ack on your post. This post was indeed intended as just "musing and rephrasing", it didn't occur to me that it might come across as dismissive.

Thank you @bobhaugen for mediating here, and mix for clarifying how you read my post. I know communication is not my biggest strength, and it is phantastic to be in an environment where more experienced people like you two keep things from derailing.

I'll try to write a proper response to the content of your actual post later, but for now I just want to get this meta-level stuff out. Especially since we seem to both be awake right now.

I think I'm more experienced in building clientside, and I think by working together we can figure out some awesome solutions.

For what it's worth, I fully agree with everything in this sentence. Even if it might not always come across as such.

If there was something else going on for you @Aljoscha that needs sayin, I'd love to hear it.

We only have finite time per day, and I spent a lot of mine these last few days on the protocol stuff. There's a lot of writing in a non-native language, and there is even more time thinking through this stuff. I have a clear sense of where I'd like the protocol to get to, but it's the details that are time-consuming. Well, and the parts where I am obviously right and everyone else is obviously wrong ;). Those are annoying too.

I did start typing out a response to your post specifically, but I didn't feel it would come out right, so I focused on some other things first. Which included not only writing posts that required less thinking for me, but also playing the piano, drawing some birds, fiddling around with #bpmux, doing the dishes, and sleeping (we live on opposite sides of the planet). Sometimes I even interact with other humans in meat-space. All of these are things I neglected the past few days.

I'm willing to put in the time the whole process will take, because I believe it will be worth it. But I don't feel any obligation, so if I feel like drawing some birds, I'll do that instead.

@mix

mmmmm.... not sure. I think matt is right about post type really benefiting from timestamp. It's really annoying reading things that are old when you want new things and a person just came into gossip range of you. or rather, it's strange, and sometimes embarassing when you don't realise how out of time your reply might be.
It's annoying when a whole persons feed sits on top of all your other friends content, that makes me tend towards blocking to remove the noise.

I think causal order would mostly solve this. Imagine if the ssb-client API had taken causal order into account from day one, do you think you'd still need a timestamp on every message for convenient client development? Implementing causal order in sbot, adding client-sbot rpcs that use it, and recommending them as the default is something we should probably do. It would be unreasonable to expect every client to do this by themselves. Checking consistency of claimed timestamps would then be trivial, even if they are part of the message content, not the metadata.

I'm somewhat inclined to dismiss every argument of kind "message type xyz benefits from timestamps, therefore timestamps should be mandatory". Ssb does not enforce a schema on your data. If you as a dev prefer to have timestamps available, feel free to add them. But if you want true interoperability, you must live with the fact that others may still produce messages without them, just as you must currently live with the fact that all timestamps you receive might be bogus.

I think if you remove timestamp you might have to make branch compulsory.
I think branch is actually way more interesting that timestamp.

I suppose you are talking about post messages here? As said above, you can just add a timestamp to it. But I agree that branch is more interesting, for the same reasons that I prefer relying on causal order rather than timestamps.

Downsides would be losing ability to do any easy queries.

<subjective opinion>If you can not rely on it being correct, then sbot should not make it easy.</subjective opinion>.

Do note that queries are implementation specific, not part of the ssb protocol. Ok, that is only half true, here is the more precise version: I do not know what kind of queries are part of the ssb protocol and which are not. Sbot uses the same rpc mechanism for talking to other servers and for talking to client applictions. Since every server could call these rpcs, they are technically all part of the network api (aka ssb protocol). In practice, a lot of these rpcs are intended to only be called by clients. For replication, you basically only need "hey fellow server, please give me all messages of feed @foo, starting at sequence number 42". There's no queries involved in that. I'd prefer if we clarified which subset of rpcs is protocol relevant, and which is not. And timestamp-based queries would probably end up in the "not part of the ssb protocol" camp.

Instead, they would be sbot-specific, and applications written against that interface would only run agains sbot as the server implementation (or any other implementation that emulates the API). And that is totally fine (for the context of this discusion), it is also how any more advanced db stuff via flume works. But ultimately, the core ssb protocol specification is not directly effected by this.

Of yeah I tihnk as the database gets older being able to say "only message published in the last year" might become really useful. I think this might be the case with all message types. Something like tags don't really need a timestamp to operate, but at the same time, if I want to see new tags my friends have been gathering how do I do that.

Causal order to the rescue again. Pick an arbitrary message which you trust to be from the correct point in time (e.g. a year ago), query for all tags which are higher up in the causal order.

Additionally, even if we don't consider queries by receive time as part of the ssb API, a server implementation is still free to store those and offer an API to the client. Combine those with causal order, and you get a pretty good approximation. And timestamps don't give you anything but an approximation as well, except that it can be arbitrarily bad.

I think we need to imagine a range of scenarios and propose how to build those quiries. Or to spend a week re-writing modules and avoiding using timestamps to see how hard it is.

This would only be a fair comparision if sbot allowed to sort by causal order. I hope that in that case, the effort for clients would be reasonable. But simply avoiding timestamps even though sbot development has assumed their usage for years, will only give a skewed outcome.

I am against anything that makes querying any harder than it currently is - either I don't have good strategies or some queries are really hard already.

Again, we need to be careful about what of this is actually part of the protocol, and what is not. A server could have non-protocol rpcs that look for a field named "timestamp" inside messages and sort by it. I unsurprisingly think of that as an abomination, but if that is what it takes to keep the core protocol simple, I'm willing to accept it.

@Aljoscha

The following are the RPC methods scuttlebot will allow any peer to call. (found by running git grep anonymous in scuttlebot, ssb-blobs, ssb-ebt, and ssb-ooo repos)

• createHistoryStream (legacy feed/message replication, still called remotely by default scuttlebot, and go-sbot)
• gossip.ping
• blobs.has, blobs.changes (legacy blobs replication, not called remotely anymore that i am aware of)
• blobs.get
• blobs.getSlice (not called remotely by any implementation that i am aware of)
• blobs.createWants
• ebt.replicate
• ooo.stream

If a peer is connected via a valid invite code key, scuttlebot allows it to call only these methods:

• invite.use
• getAddress

Thanks for taking the time to break your thoughts and response down clearly @Aljoscha. It's also really good to be reminded that I'm in my mother tongue, and that you also have other balances in your life which are important, thanks for those

Here's a few reflections and responses on what you've said:

I'm somewhat inclined to dismiss every argument of kind "message type xyz benefits from timestamps, therefore timestamps should be mandatory"

We're on the same page. I wasn't suggesting mandatory timestamps just cos posts might benefit from having them.

I think if you remove timestamp you might have to make branch compulsory.
I think branch is actually way more interesting that timestamp.
I suppose you are talking about post messages here?

No, there are a range of message types which need causal sorting - e.g. when edits happened on a gathering, or different responses to a poll came in. Different sorting creates a very different meaning. Without a branch property (and assuming we remove timestamp), it gets very hard to arange things causally.
I guess this could be optional, but if we agree causal sorting is a good thing to be able to do I'm nervous about making many of the ways you might do that be optional (e.g. by having no requirment for timestamp nor branch anywhere)

Causal order to the rescue again. Pick an arbitrary message which you trust to be from the correct point in time (e.g. a year ago), query for all tags which are higher up in the causal order.

I like your distinction about the protocol level versus the client level, that's helped me reframe this conversatin more. I can't currently imagine how to make a way to write causal queries on the server. I'd have to think about it... at the moment it would have to be an index, and because we don't know when different messages arrive ... hmmm. A tricky problem.

I think there are two possible futures:

• people mull on that problem and try to find a solution which is performant
• we don't find a performant way to do causal queries, which <opinion>forces use to defer removing timstamps / branch</opinion>

@mix Any (non-garbage) cypherlink in any ssb message establishes a happens-before relation. Most messages in the current social networking applications include cypherlinks. Building the partial order won't be a problem, and it is unlikely that it will be too sparse either.

By adding cypherlinks as a dedicated logical datatype, checking messages for them will become very easy - with wort-case time complexity linear in the message size (with far better constants than would currently be required), and a lot faster in the average case where you get to skip a lot of data of differing types.

Implementation of causal order in sbot would be a great way for anyone to contribute to the current push to renew the protocol #somebodyshould. The causal order is simply the reachability relation on the directed graph obtained by treating ssb messages as nodes and cypherlinks as directed edges. The linked wikipedia article mentions a few algorithms, and there should be a ton of literature out there. The implementation can stay fairly simple (without changing the database representation), but in the future it might be worth considering to adapt the db to allow constant-time lookup/computation.

Fun fact: Combining the causal order aka reachability relation with some sort of tiebreaker yields a topological sorting (which is a total order).

Computing the causal order is another of those things where all the thinking has already been done for us. Moving to a db with constant-time reachability lookup will be some work, but might be worth it. But it is not required to so before we roll out causal order sorting in sbot, there are less sophisticated but still sufficently performant algorithms.

CC @arj, @cryptix and @keks for database related thoughts. You might want to keep this in mind for the go implementation.

nice summary of causal sorting.

I knew how it worked, and am interested for us to test if there's enough edges to get a decent sorting without requiring more. My gut says there might not be, but that's just based on my work with branches and writing some tests around ssb-sort (I wanted to be able to display messages which "didn't know about" other messages ... as in they were written in parallel, without context, as we display threads linearly, when in fact they are not always so linear).
But it's an empirical question.

I also have gut feeling that the current flume indexes would have a hard time creating a decent causally sorted index...I feel like that would need a whole thread in itself.
I am totally drawn to the idea of a causally sorted database. It would be resistant to time drift and (maybe) relativistic time dilations (SQUEEE!!!)

Unless I missed anything, this means that the server-to-server communication only relies on sequence numbers, and is not even aware of timestamps. So optional or removed timestamps will not directly effect the non-implementation-specific rpcs. That's pretty great.

TODO: At some point, #somebodyshould clearly document the boundary between server-to-server and server-to-client communication. This is exactly what implementors need to know to get a minimum viable pub working.

@mix:

we don't find a performant way to do causal queries, which <opinion>forces use to defer removing timstamps / branch</opinion>

I tentatively agree with that opinion. But what about making timestamps optional? How would you feel about optional timestamps without support for causal queries (yet)? Because it looks like the whole database reachability thing will be difficult enough to push into a later update.

Unless I missed anything, this means that the server-to-server communication only relies on sequence numbers, and is not even aware of timestamps.

Definetly the case, at least for non-ebt replication. I'm fairly certain ebt also doesn't as well, otherwise @dominic wouldn't have given in on non mandatory. Legacy replicate really only involves one rpc method and that is createHistoryStream it's main arguments are id:pubkey and seq:int to specifiy the starting point of a particular feed. This is all we do in #go-ssb and it works like a charm, never had to pass a timestamp somewhere.

Optional timestamps is probably fine, but will force a bit of rework across a bunch of things.
Like I said before I think I'd like to be forced to try doing that before removing them.. maybe we could set a deadline to do that by?

@mix The minimum change would probably be to just ignore all messages without a timestamp. This reworking would then consist of finding all places where things would break without timestamps, and dropping the message instead of handling it. Anything more advanced is completely optional.

Do you have any preference on how ssb-client-js should signal an omitted timestamp? I currently see two reasonable approaches, either completely removing the "timestamp" key from all message objects without a timestamp, or keeping the key but setting the value to NaN. Any thoughts on what would make client adaption easier?

CC @Matt McKegg, would you also be up for looking through the patchwork code base for places that break when timestamps are omitted or NaN?

@Aljoscha

would you also be up for looking through the patchwork code base for places that break when timestamps are omitted or NaN?

Sure.

Off the top of my head, I think that updating this patchcore helper would go a long way to handling this case (but there are still a lot of direct calls to value.timestamp).