You are reading content from Scuttlebutt
@aljoscha %dn7ucXaeyjozlAE2Ma5gsPd2mpi8an4hmgGGk83QaqE=.sha256
Re: %iy6Pekirb

Getting out two more thoughts on metadata: One on the type (as in "string", "map", etc.) of message content, and one on the actual implementation of optional timestamps. Both are somewhat connected by their usage of a varint tag for metadata.


As alluded to here, conceptually there really shouldn't be anything forcing the content of a message to be a map. Nothing makes maps inherently more special than other json/hsdt values. But with the current ssb design, there are two aspects that restrict the type of the content field:

  1. private messages always use a string ending in ".box".
  2. Each message must have a type, which is currently stored in the content - that's the thing that forces content to be a map for unencrypted messages. It also forces the encrypted string of a private message to decode into such a map.

The first point is a somewhat hacky implementation with some unintended consequences, a metadata varint tag can clean that one up. If the tag indicates that the message is encrypted, it does not have to reuse the same metadata structure as unencrypted messages, it can simply include the cyphertext (or its hash if we implement offchain-content) as a binary string.

The more interesting case is the type field. Since each message has a type, it is conceptually part of the metadata. But the type should be encrypted for private messages. But those only encrypt content, not metadata, thus the type was moved into the content. Which - again - is more of a hacky shortcut than a clean solution. But again, tagged private messages allows us to change that. Encrypted private messages can encrypt the type and yet keep it in the metadata. And with that ability, regular messages don't need to put it into the content, instead it can be moved into the regular metadata (note to future me: argue to make type a binary string and to increase the size limit to be able to contain 512 bit hash digests).

And with those changes, everything is disentangled, and the content is not arbitrarily restrained to be a map.


The other point are timestamps, in particular their optionality and their privacy. This part of the post makes the assumption that optional timestamps are allowed.

As I already wrote about above, private messages do not encrypt metadata. And even though ssb explicitly hides the number and identities of the recipients of a private message, it happily attaches plaintext timestamps. That's kinda weird. By the same mechanism that can encrypt the message type, I propose to also encrypt the time stamp. A non-recipient of such a message could not tell the timestamp of such a message. Currently, this is a problem, because ssb assumes that every message has a well-known timestamp. But if we are in a setting were timestamps are optional and private messages encrypt them, non-recipients can treat all private messages as timestamp-less and no further changes are required to any code paths.

Now for the concrete representation of messages without timestamps: I'd simply use the message tag varint again. Messages without timestamps then simply don't include one, saving 8 bytes. Even if an actual API like ssb-client might want to signal absence of timestamps via a garbage value, there's no need to send any garbage values over the wire.

Two more quick, unrefined thoughts on these timestamps:

  • should an encrypted message hide wether it is timestamped or not? If that is signalled by the tag byte, than that information is public. I don't really see how that would hurt though.
  • users should be able to set whether they want to attach timestamps to their messages or not. But in some cases, devs might want to have full control over timestamps, so they'd move them inside the message content. If they do so, is it a good idea to allow them to signal that no metadata timestamp should be included? Saves space, and more importantly will prevent a lot of ambiguity and weird user interface behavior if metadata timestamp and internal timestamp differ.
Join Scuttlebutt now