Schema Versioning

#dev-diaries #mmt #darkcrystal #dark-crystal

Since we're looking to extend the functionality of Dark Crystal, this is likely to mean that we're going to come up against a switching over of schema versions. We want to avoid breaking changes to already existing dark-crystal records in our database, so @peg and I this week looked at implementing a schema versioning system, which could be transferable to other ssb modules (such as ssb-poll-schema). I know @mix and @pie pete implemented some version of this in ssb-poll-schema but I'm not sure how far they got! Anyway, I hammered it out yesterday and this morning. Its a bit hacky, could do with a refactor, but it works.

Pull request for ssb-dark-crystal-schema

Each message in the dark-crystal schema stores a version number (as a string "1.0.0", "2.0.0"). So in our validators, we now check the message version before anything else in order to retreive the relevant schema, then we check whether the content of the message is valid. This was a bit tricky and haps to have got it working (tests pass! whoop!). It was a good step to take as it was going to be an issue at some point, so felt better to tackle it now than having it land on us later when we're short on time / funding.

I also built out the dark-crystal/forward schema while @peg worked on the API for publishing and getting the correct messages.

Taking into account Dominic's recent points in his security review of dark-crystal, its possible we'll need to have scuttle-dark-crystal behave differently depending on message schema versions too. Something we'll have think about if we make any breaking changes to the API.

had a quick read this morning on my phone, looking forward to being able to test it more thoroughly when I have some time @kieran

I refactored my changes into a stand alone module for verifying ssb records with is-my-json-valid called ssb-schema-validation, which might be useful to others #ssb-show-and-tell