%ZBzVJhKOYub7bFtZpYCeXPa3O7Me4gftEl80b+ts6Kk=.sha256This thread will serve as my dev-diary for the #sunrise-choir. General goals:
%SNbftBoc4Mbq22hbezl5goKOEfoAiHTPN6avqvwRLAg=.sha256Dev Diary 01/10/2018
TODO: Check whether float handling is really what nodejs does. I just applied NOTE 2 of the ECMAScript spec without knowing whether node actually applies it. I also specified the rounding for the
s * (10 ^ (n - k)) === mconstraint to be round-to-even, without checking what node does, and without knowing whether rounding can even occur. Will check this once there is a test suite, or maybe I'll just ask the v8 people.
Next steps:
Deal with floats (I really hoped I wouldn't have to, but oh well...):
Next steps that will be actually useful:
But yeah, tomorrow's dev diary may very well end up being "I read some papers and scribbled down a lot of notes trying to understand them". Hopefully it won't be more than one day of that.
%ezmzK85DZ5mbG2bQcIFEl7a1kam1IPLM9FCSBfmmq8Q=.sha256Dev Diary 02/10/2018
Next goals:
Bonus goal:
%AFLMLHLG/ALnOgljKlPN/AlAfTFO9vfOfmxJ52Q4TM4=.sha256Dev Diary 03/10/2018
U+FFFF by doing two four-byte escapes and treating them as a utf-16 surrogate pair is fun! Unicode is Fun! Json is a simple format!I haven't run a single line of this code, but in theory we can now encode and decode arbitrary json legacy data values.
Next steps:
%8C4muzwvOywlOEeK4UFgsikK5OT8+Tk9BdkGrcX+gsk=.sha256Dev Diary 04/10/2018
Today was somewhat grindy, chasing a lot of errors.
Fuzzers are magical, this is all the test code I wrote to weed out errors in the json parser:
#![no_main]
#[macro_use]
extern crate libfuzzer_sys;
extern crate ssb_legacy_msg;
use ssb_legacy_msg::json::{from_slice, Value, to_vec};
fuzz_target!(|data: &[u8]| {
match from_slice::<Value>(data) {
Ok(val) => {
let sign_json = to_vec(&val, true);
let redecoded = from_slice::<Value>(&sign_json[..]).unwrap();
assert_eq!(val, redecoded);
}
Err(_) => {}
}
});
fixed everything the fuzzer found in a few seconds of running: https://github.com/ssbrs/legacy-msg/commit/f2cb1656d2bcb3283a6775ddd30e2cfac745c5b0
the fuzzer found that f64::from_str erroneously rejects some long numbers such as 11111111111111111111111111111111111111111111111111111111111111111111111111e-323 (which can still be precisely represented as an f64). Guess I'll now have to look into float deserialization...
strtod, because why not: https://github.com/ssbrs/strtod/blob/master/fuzz/fuzz_targets/fuzz_target_1.rsNext steps:
%L3sTXRyH0N20R4624x1ebJigtn1zMmsvxXtXrcZN3HI=.sha256hmm is JSON.parse("11111111111111111111111111111111111111111111111111111111111111111111111111e-323") equal to 1.1111111111111111e-250? because that is what node.js interpreted it as
%qHIsyGax1AV3DNPXF4Cjyd7dvqPgPSl8fENxKc6xDoI=.sha256Yeah, those two are equal. Parsing it may actually result in a different float, but rounding works for both serialization and deserialization.
%TupGHyN+zrXoLesJsmf5fm0pib/kxMH10I3BWjwbL+k=.sha256Dev Diary 05/10/2018
This was not a fun day of work. The most productive thing might have been the fuzzers running in the background =/
%O8aEFQ3FJQir6ogLs5KA/+BDM+NnceR3eIbzXnlCBEA=.sha256Dev Diary 06/10/2018
(Not a full day of work)
Next up:
I'll do the cleanup, thorough documentation , and actual releases, once both json and cbor legacy message data are fully specified and implemented. I'll tackle legacy metadata afterwards. And when that is done, it's time for #hsdt
%AlSseDWFG+UcKCVyPFXPVMSl84YV17FF9yQFldH6Nyw=.sha256Dev Diary 07/10/2018
%KpresJXwd6VojMcDUFDDFC6c0sU/40CUIIabX+peK/k=.sha256Dev Diary 08/10/2018
Next steps depend on how we deal with the signing problem and the cbor feed id encoding. In any case, I'm close to having implemented the full legacy message code in rust, both for json and cbor. When that's fully done and we settled on how to deal with the open problems, I'll do a bunch of clean up and publishing, as well as js binding.
%qXZR406vxMxymxd2jC5g0P2Ij+oOKBlA5XnwtSuM17I=.sha256Dev Diary 09/10/2018
Today was another of those grindy days. The above doesn't look like I even did a lot, since most of the time went into reconstructing error cases, fiddling around with the rust standard library, and debugging.
But on the plus side, this may have been the last grindy work on legacy message stuff. Json encoding is fully implemented, the precise format is nearly nailed down, cbor encoding appears to be working, and I think I've settled on a compact legacy metadata format to go with the cbor content format (I'm pretty convinced that this is the right way to go for feed metadata, which was the last open problem with the compact legacy metadata).
What remains in this first phase of work is code clean up, documentation, better cbor testing (ideally by checking against other implementations), better cbor spec, specification and implementation of the compact legacy metadata, clean-up of the spec, release of test data, binding creation. All of these are tasks I have full control over, no more bugs in dependencies or protocol weirdness.
I expect to do the first releases of spec, test-suites and rust implementation later this week (leaving cbor stuff for later) 
%dAsf1LY06v2KAOdTf6jAEsQTeivQw1ha8az5Ecnk44c=.sha256Dev Diary 10/10/2018
This felt unproductive... oh well.
%PW43Ppg8fC29pd0SN4wX35vv1QZOgWRzFkpMjuKY4Lg=.sha256Dev Diary 11/10/2018
%GKStctEutueJ14zwLhVO69cMCspyfLfmtlkeuQmjycs=.sha256Dev Diary 12/10/2018
Next up:
I won't manage to finish it this week, but reasonably soon I'll be done with the legacy stuff. And then I can start thinking about protocol improvements again.
%Ry/WwEavJIGalPiDh3AWTvX1rP9VmcMlcvl7CrhIyFU=.sha256Dev Diary 13/10/2018 and 14/10/2018
Spent a lot of time fighting the compiler and my own traits. I got pretty frustrated, it has taken me ages to write a tiny amount of low-quality code for the metadata implementation. But I think I figured out why: I've misunderstood serde. I wrote a serde-like crate where the supported data model is exactly that of ssb. But the serde data model isn't about json/cbor/whatever, it really is about the rust type system. Data format specific stuff can be done by the Deserializer/Serializer implementations (and most importantly their error type). This was why writing code with my custom traits was so painful - they simply didn't map to the rust types I tried to (de)serialize.
The moment I realized this, I went from having to force myself to do more rust implementation work to actually looking forward to it. But I'm done for the weekend.
Other things:
I can already feel this dev diary declining...
%IvA7g15L0VzwVkDC7qQtoErigUfnl25V7yQ3wLNdUEo=.sha256Dev Diary 15/10/2018
The problems with serde still persist, there are a few things in the ssb metadata that serde just isn't equipped to deal with. I don't have the energy to push through this right now, I'll take the day off and discuss this in the next sunrise choir call (which luckily happens tomorrow).
%a/sPmkGYYAkpw2pFWo3i7PBsXQ7Q4xM5CrBzTl0U9EI=.sha256Dev Diary 15/10/2018
%a23cZ8SH7UDMoU65Et0gV8bPGLsb4IvUg0z8CgvlbR0=.sha256Dev Diary 17/10/2018
%ymulqy/SgRxO/Cr3YsGomb6FWC9I10FYj2qSKZXtqL4=.sha256@dinosaur, can you do some magic so that this gets compiled and published at spec.scuttlebutt.nz upon a push to master?
@Aljoscha sure! 
%uFwD/xs0rH0mzP7CbQC+lMFa1HVjUh1FLNA/JJwFe6Q=.sha256Dev Diary 18/10/2018
Nothing to report, just a bunch of code that needs to be written. Doing both in parallel, neither is done yet. Code is in such a terrible state that I won't bother uploading it.
I'll continue working, but I can already post this, nothing new is going to happen over the next few hours of work...
%oJ9A++vrbxXlCgGLqttaVh5eFmQBA/rqpkBiq5QXHkY=.sha256Dev Diary 19/10/2018
merkle skip list would lead into such an amazing rabbit hole%irDHnh42eR+I94+MLq9JMCwQMjOOmlXJZ8OhIPD5uM0=.sha256Dev Diary 20/10/2018
Everything is ready to get back to metadata handling. Once that's done and base64#76 is resolved, I can generate test data sets, and then the legacy stuff is over.
%Uqc0CJeEjJ86nbRjS2MAoSoHTWkFB0Cik06T8HGugQs=.sha256Dev Diary 21/10/2018
%x5jH4DElARkUWnaalO9q6q7L8NVmPc4z201QcRarWc8=.sha256Dev Diary 22/10/2018
Writer.%aaedvtZsTkRF8mFzdjNwG4u3Aoi5a5pOczeganXq4Qo=.sha256Dev Diary 23/10/2018 and 24/10/2018
%S9Nt+Hz0BsfI3EkicwbLjppRma+sLMfx31Qp+ruGO/0=.sha256Dev Diary 25/10/2018 and 26/10/2018
The progress feels extremely slow, I have reached the 20 days of work per month (with a lot of those spent completely on sleep, food and ssb), and there's some rl stuff occupying my mind. So I'm going to take it slowly for a couple of days and work mostly when I feel like it, rather than my previous, pretty rigorous schedule.
%TQye2G14ksYc2ked7hFDtQaFlQAKsGHO1dKxBvlUUAI=.sha256A little bit of clean-up, and then the legacy message stuff is done (well, except for the js bindings).
%P/+NydXsVHg/9xnXTqrJbYsjBWv3EGI//W1XeXDfn3c=.sha256Dev Diary 29/10/2018
%58xHLWdOGKG79KSxMS1LBJrG7INse0TGO6BXcktRLlc=.sha256Dev Diary 30/10/2018
%aER5wuLkfTF5LvwwgjRKaURp94/1oDbzR9WzjnzRVd4=.sha256Dev Diary 31/10/2018
%PSFz5I4macpmCoNtMtZalrdpD9f95WzZCUoJJFDfT68=.sha256Dev Diary 01/11/2018 and 02/11/2018
Life outside ssb has been somewhat draining these past few days, so I haven't had the energy to engage properly with @keks on multiboxes and @Tim Makarios (on Manyverse) on merkle trees. I haven't forgotten either of you, I'll get back to these discussions once I have the energy.
Note to self: Start discussion about clmr's representation of hashes (explicit sigil or not?) and feed ids (data other than key or not?). Clmr just takes an arbitrary stance on both of these, but we should come to an informed consensus - and I'm honestly not sure which way to go.
%ORYTV6aFYAOpiMRwr7XJjmJEr7R9COGgzdNZVf1ImbQ=.sha256%Ylf2GxoMLhh5jH1pV8akERmnertAwvFDuEVE6FaXyaQ=.sha256Dev Diary 05/11/2018
%Y0MKglyq4wkV10GqEsrMSZOXNLZnaMdUJ43eT7NnlrQ=.sha256Dev Diary 06/11/2018
This felt pretty unproductive, but it was a day of doing stuff, so...
%JbM1uozrpLjBfF4TwvNFnWXl9MNFVKWrdj5YRxahtVo=.sha256I always imagine it as pouring water into a pool... you might not see it, but the water level is rising and each cup counts :)
%eYszHVzArgPhEdV3OUzqbgyR1FAUhQPOUrKhyLw3IYw=.sha256Dev Diary 07/11/2018 and 08/11/2018
%oS2l3S2sYWlcQxAOtXJO5r5LJ5VqnkFTq+ZlAWaz8oo=.sha256Dev Diary 09/11/2018
Had my first fights with the borrow checker for this project. State machines that produce some sort of result are annoying in rust: The functions to advance them work by taking mutable references. But when they are done, you want to return the result you have been accumulating by value. The workaround involves putting the accumulating result into an Option, so you can take it at the end. This introduces a bit of runtime overhead and makes the code noisier, but it satisfies the borrow checker.
Technically, the borrow checker is correct, because the type system does not express that the state machine is never used again once the result has been produced. But still, in C you could just write the straightforward code and everything still works as long as the state machine is used correctly. And it is not like the Option-workaround would prevent incorrect usage, it just panics at runtime.
The whole bpmux/rel implementation will consist of composing state machines, so I'll just have to accept this.
%RIjXvqNgVkkg6SvZTJHgIBMtJlJwEcWUE0UX9kyPr84=.sha256Caught a cold, it'll be a few more days without updates.
%VQ33qhvPghu4MwzSzt8+lYRO1XpDlxVJofIqWZOcEZ4=.sha256cypherspace bed of restoration for you friend! rest up and take good care.
%OQoUFhCmp7/EH/Lgxi5C1TYf68xNkCmnsy2lLs7h2KU=.sha256Dev Diary 26/11/2018 and before
I'm mostly back. The cold segued into a phase of really low motivation, and I likely won't get back to my previous schedule of work on all the things all the time, so expect more sporadic/irregular updates. Had a good, open conversation with mikey and piet over this, would definitely recommend them as boss/coworker.
Aside from sunrise choir calls, I also had a call with Dominic where we talked about the future message formats. I'll finish the clmr format, then I'll focus on specifying and implementing hsdt and the new metadata format.
Stuff I did today:
Misc:
Still need to implement the new multibox definition, but then the legcy/clmr block of work should be done.
%WC2/Y6508efp3LxtDXsJ4hZlj4HPMRXWMKuN4zn1dGg=.sha256%I0vGZNgEBWRyninIHVF/F9ZItu46vK8QShIj6ogrn2Q=.sha256Dev Diary 03/12/2018
This should fully conclude the work on legacy messages. We have a full spec, rust implementations for deserialization, validation and serialization, and a more compact encoding (both a spec and a rust implementation).
Unless the multibox definition changes yet again, the next block of work will be the final definition of #hsdt (and a rust implementation, as usual).
%SlfGcmglfK+kxu+YnEWpqVzPTXButru/5Uxd4jYMnb4=.sha256Dev Diary 04/12/20
%svxxo2D1B9li5ka+yJslVk2vCyitEb0HWwnfL8LLyKs=.sha256End of Diary
This is difficult for me to post about, but here we go: I've left the #sunrise-choir team. I managed to burn out, there's been an unhealthy cocktail of:
I'm starting to do better, but I need some time to recharge. I've started out being very passionate about the protocol discussions, but right now I mostly skim them and move on, even if I very much disagree. That's pretty scary, but I'll listen to my brain for now. There'll surely come a time when I'll engage again and can annoy Dominic a bit more.
@dinosaur and @Piet have been absolutely lovely the whole time, while still giving me the freedom to learn this lesson about myself. Thank you guys.
I plan on writing a retrospective on the stuff I did over the two months of choir work. There's been a bunch of specification writing, both for the current protocol and for potential protocol evolution, as well as a few rust implementations. A few things are somewhat hanging in the air, but a lot of them are self-contained and usable. It'll be nice to organize and reflect on them. I'm not in the right state to tackle this yet, but I know I'm going to get to it eventually.
%C58RKMM/c7GeY3kNEX8DKbL2mtI8KtM7Hx/N6OMl4dQ=.sha256%NkE0di9XuWXMFt7NdzQ/3U/3yXEIFBAMt3X1uVkQsTk=.sha256