@Dominic I'm all for backwards-compatible extension and against a reboot/fork. I think hsdt can be introduced in a backwards-compatible manner.

Just to check whether I correctly understood your formulation of the backwards-compatibility criterion (it took me a few attempts to parse this):

The first part of your formulation of the backwards-compibility requirement ("It should be possible to represent the current JSON based data in the proposed format, [...]") does not talk about the logical data models, but about concrete representations, correct? Because the logical model is satisfied by hsdt: The logical data model of hsdt is a superset of the js values that form the logical value of json. This distinction is important, is important, e.g. json does not support Infinity, yet the logical model contains infinities, since e.g. 123E456 gets parsed to the float representing positive infinity.

But for concrete encodings, it's true that canonicity and this criterion conflict (for object order, as well as for floats and for escape sequences in strings). But do we really need such a strong criterion for introducing the new encoding without forking the network?

I simply assumed implementations would keep sending messages in the encoding that is used by their hash. So all old messages would be sent as json, but newly generated messages signed by hashing the new encoding would be sent in this new encoding. There'd be a transition period where sbot would update to include code for dealing with the new format, while still using json. And once sufficient time has passed and most sbots are able to handle the new format, we start using the new encoding.

We don't get the efficiency benefits for old messages, but this is playing the long game. And if you want rpcs that can transmit old messages more efficiently, that's completely orthogonal. You could implement rpcs that transmit old messages as non-canonical cbor, the standard even contains a section on how to convert between json and cbor. But this does not effect the encoding used for hash generation at all.