@mikey6 years ago %dtq8ag0+ywIa4qGyTMF7rE7xFpMQJxDvg9JvOK/CtS4=.sha256

#sunrise-choir notes

here's a thread for the (Rusty) Sunrise Choir to share notes from our team meetings.

if you want to start a conversation about something in the notes, please fork into another thread!

@mikey6 years ago %ac17jOfdlLP93R2dTp2u3zQALiOKeUDCQAXhA3zTX6E=.sha256

#sunrise-choir

sunrise choir - 17/09/2018

attendees

mikey
piet
aljoscha

agenda

checkins
???
what's our team rhythms?
what processes do we want to have?
- in previous teams
  - what did you like?
  - what did you not like?
- how often do we want to meet?
- what do we want to do when we meet?
where do our things live?
how do we want to communicate?
recap roles and expectations
- how do we want to be held accountable for our work?
- what do we want to do?
- what don't we want to do?
sketch out first phase of work
- what's the surrounding work that isn't dev?
rust best practices?
- do we have guidelines?
???

previous team experiences

piet
- enjoyed having social elements
  - catching up in the morning, see how people are going
- hard to deliver in a mechanical way
  - don't work well as a machine
- tend to like a thing, work on it, then put it down and focus on something else
- like kanban-ish board to get overview of tasks, have discussions, sort by priority, even if not organized into sprints
aljoscha
- first time working as a programmer
- prefer a batch mode, get into a problem, obsess over it, figure it out, until completely done
mikey
- first time being a project manager
- freedom to go deep, or pull away
- burn out comes from not being able to pull away

rhythms

"daily" dev diary
weekly meeting
monthly retrospective

processes

kanban-ish board
use GitHub for our own things
- have ssbrs + sunrise-choir

accountability

piet
- accountability
  - at start of month, say how much gonna work
  - at end of month retro, how many hours did you work, did you do your dev diary?
- roles
  - hesitant to write specs, but think we really need to write specs, want to get better
  - keen for rusty programming
aljoscha
- roles
  - not hesitant to write specs
  - like programming in rust
  - getting a coherent picture of the whole stack out of people's minds and into a document
  - thinking about how to write a really accessible implementation, what needs to be there, what doesn't need to be there
    - do we want to be a reference implementation?
    - accessibility vs performance
      - idiomatic rust doesn't care about running out of memory
      - lean towards accessibility, can always do performance later
- accountability
  - not sure yet, we can bookmark at first retro
mikey
- roles
  - meta-team things
  - don't want to do project stuff
  - do want to do more governance stuff for rs and this
  - do want to be available for peering / helping etc.

stack

piet
- flume
aljoscha
- core protocol design

what is mvp?

@mikey6 years ago %2PNzqbmXzw8yUwlleMac56Spx44QJfw2Tu8ozRUnNVY=.sha256

#sunrise-choir

sunrise choir - 27/09/2018

attendees

mikey
piet
aljoscha

agenda

checkin
sunrise-choir / ssbrs
- alj: like clear distinctions between specs and rust code
what aljoscha will start on this week?
how do we add new things and get @dominic's approval?
neon / n-api
async monads
reachability in the database
plugins

aljoscha start

define the logical data model of ssb messages
define rust traits / abstract apis for dealing with them
- serde
specify and implement legcay json encoding
transitional codec for transport between servers
- subset of cbor
- should this be used for signing?
  - yay: json codec is ugly
  - nay: if we introduce a temporary format and use it to sign / hash messages, it will be around forever
start an official ssb spec
- a hard spec, less fluffy than protocol guide
- able to drill into any part of the protocol and dive deep
- not a linear document from start to end

how many hours in a work day

mikey: high trust hand-wave

how to get approval

recent plugin convo: %UyEnqEk...
mik: dominic is worried about us being all specs no code
- maybe trying to nip a divergent spec in the bud
- a perception where we create more work and not do the work
  - even if our perception disagrees
mik: dominic likes negotiation
- choose our battles
- be clear about what is high priority
- be willing to let go of what is not priority
pie: how to explore the problem space without starting an argument
- maybe make it explicit
mik: swallow bits of javascript with native rust
mik: good to build momentum and trust
pie: get on a call with interested parties and make a case
- alj: high-bandwidth conversations are not accessible
  - pie: still has value
  - pie: prioritize notes
- alj: try to do as much as possible in text, not calls

neon / n-api

neon looks stale

n-api looks promising

async monads

context: %HjMfx1M...

alj: rust has good generic interfaces
- can test against the interfaces directly
- macros for property-based testing of reader / writer
alj: now rust supports afl fuzzer

reachability

context: %uJX/grT...

alj
- is one thing older than another?
- messages are nodes, cypherlinks are edges
- is one node reachable by another node?
mik
- realized value of timestamps: "only fetch content from last month of messages"
- keen for the best of both worlds
  - timestamps are human
  - but time is actually causation

@aljoscha6 years ago %6uhFP/GRaWgce0kTt33OAZvDcEaCrX1NiG5ueHHc6qI=.sha256

#sunrise-choir

sunrise choir - 04/10/2018

attendees

Aljoscha
mikey

agenda

regular time slot for calls
licensing (agpl vs natifying sbot (MIT))
ssb spec format and hosting (stable hyperlinks)
feedback: spec wording and code review
overview of rust impls, part of spec or separate?
binding creation documentation/examples

why timestamps bother alj

alj: ssb is subjective, new concept!

alj: why does everyone insist on objective timestamps?

alj and mik agreed on a monthly contract

best way for alj to legally do work.

5k USD per month.

alj will try to do 20 days per month.

regular time slots for calls

mik: keen for monday, tuesday, wednesday evening nz time

alj: flexible, want something regular

alj: would like if started at 9am berlin time (8pm nz time)

licensing

alj: current rust code is AGPL. sbot is MIT.

alj: if we rust-ify sbot, then sbot would need to be AGPL too.

mik: this might really be a question of governance: what is our (legal) strategy with respect to Scuttlebutt code assets?

mik: for now, maybe stick with AGPL, but ensure that those who contribute (copyright) are willing to change in the future if necessary.

ACTION: mik will start a conversation about this

stable hyperlinks

alj: want to refer to section -> subsection -> blah by url

Put more abstractly: Want to refer to specific ssb concepts in a machine-readable way. The web is just an (obvious) implementation detail.

mik: could offer spec.scuttlebutt.nz or spec.scuttlebutt.rs

what's domain do we use?
- scuttlebutt.rs?
how do we make html?
- GitBook for now

code review

alj: want to convey good overviews and understanding.

alj: want feedback.

am i being precise enough?
do i cover everything that needs to be convered?

alj: leaning towards being as precise as ~~possible~~ necessary.

rust impl part of spec

alj: should we link to the rust impl from the spec as a reference implementation?

alj: a story of three parts:

spec
implementation
test cases

alj: should we link them all together?

mik: yes!

should we document for binding creation?

alj: for example: how do i let the parser decode into other native objects?

alj: should we maintain bindings?

mik: yes we should maintain JavaScript bindings, but can happen later when we try to integrate into jsbot

mik: for example: https://github.com/pietgeursen/private-box-native. what's especially interesting is the use of prebuild so people can use our rust native modules without a rust toolchain.

User has not chosen to be hosted publicly

@aljoscha6 years ago %7bI0b8/4LHOQFt5tk/EToN1NzwqH43UrppZB+FaX3/8=.sha256

#sunrise-choir

sunrise choir - 16/10/2018

attendees

piet
aljoscha
mikey

agenda

checkin
schedule the meeting for a regular time
- Tuesday 8pm NZDT
- Tuesday 9am Berlin time
- Piet will make gcal invite for ~~Mikey and Piet~~ normal persons
schedule the retro
- Tuesday 30th of October
- Piet will make gcal invite for ~~Mikey and Piet~~ normal persons
piet would love to hear about alj's fuzz setup
review
- everything is horrible (with serde (and ssb))
- mik still hasn't started discussion about licenses
what is the process for js bindings?
what should alj do next?

review

Aljoscha

full spec of legacy message format and metadata
- includes edge cases of non-canonical formatting
- oppressive amount of edge cases
not everything is perfect and generalized
- logical is right to generate test cases for fuzzer
- once we generate a server executable, can figure out what needs to be improved
read in data, validate it, output in compact format or signing format
- hash computation: ssb didn't specify encoding, node defaulted to latin-1 encoding, in practice means encoding as utf-16 and taking only least significant bytes
metadata
- alj underestimated the protocol
- defined multiformats
- most annoying is how metadata itself is structured, serde is not good at parsing it
  - content can either be arbitary json object or string
  - serde cannot elegantly handle this case
  - solution is to ignore metadata as being json at all
should we use custom-serde or serde
- normal serde gives us potential for static traits for ssb message types
- custom serde more accurately reflects the data model of ssb
rust traits are less flexible than desired
- ideal is serde parser can directly build up a native js object
  - https://docs.rs/json/0.11.13/json/enum.JsonValue.html
  - https://docs.rs/napi-sys/0.1.1/napi_sys/enum.napi_valuetype.html
testing & fuzzers
- integration to rust toolchain, cargo fuzz
  - https://fuzz.rs/book/cargo-fuzz/setup.html
- fuzzer analyzes paths to your code, tries to find the full state space of your implementation, tries all the fuzzy bits
- test all the possible (corner) cases!
- next step is to try all the possible cases in js
  - separate what is valid or invalid
- creating great testing data sets

piet

first day, did setup, thinking about where to go
jumped into making private-box-rs have usable js bindings
as of yesterday, got js binding working
- project is not as tidy as wanted
  - faffed around with getting everything working
- will have a think about using napi-sys crate
spent today looking at how everyone else manages prebuild binaries for things that need binaries
- how does cross module work?
  - can we use his docker image? yikes rabbit hole
- discovered appveyor
- use travis ci for building binaries
- trust module: rust library for building rust bindings for many os's
  - japaric is fucking prolific and makes amazingly useful stuff
looking at how to do async decryption
- oh no, c++ void pointers and madness
- will sleep and see how ugly the story is
how to tidily manage when things go wrong
overall happy with n-api, it's the thing we should be trying to use
- only stable in node 10, but legit, is successor to nan
tidy tidy tidy
goal is to do a good job on this module
- so will serve as a good easy-to-use template for other modules
alj: is there a check whether a given string is a valid box?
- wondering whether need to implement this higher up the stack
- maybe good candidate for fuzzing

what should alj do next?

fun!

what is the process for rust-js bindings

rust-sys ?
- probably the best way to go
- pie will what he can do
pie will write up a document on how to make rust-js bindings

@aljoscha6 years ago %EdG4zcG7RWrcNju58cSfFmSGBPNfPPLic1VfUS1XxJM=.sha256

#sunrise-choir

sunrise choir - 23/10/2018

Attendees

Aljoscha
mikey
pie

Agenda

rfc

alj: who decides?
- is it whoever happens to implement it in sbot?
- mik: governance!
alj: don't care about format, as long as consistent
mik brought it up in private thread:
alj is close to making new things, so could use an rfc format soon
mik: will be the case we need to hand-wave as we go
mik: we have the social capital, as long as we are intentional with how we spend our social capital
pie: hard to say no to working code and documented systems

state of bindings

Looking good.

private-box spec

alj: pie you said you want to practice writing specs ;)
pie: mik is this a good idea?
pie: where do specs live, how should they be written?
mik: alj made an mdbook, a rusty gitbook
- https://github.com/sunrise-choir/spec
mik: i made travis auto-publish to
- https://spec.scuttlebutt.nz
- https://spec.ssb.nz

spec in general

licensed as CC share-alike
alj: want to link to stable identifiers for concepts
mik: structure is going to change anyways
mik: glossary of all stuff you want to link to

licensing stuff?

mik still needs to start the conversation
new learning is Manyverse needs copyright assignment or something similar because it can't publish to Apple Store as (A)GPL
- in publishing to Apple Store, they require you to accept a new license, which is incompatible with (A)GPL
- solution is if you own copyright (or have it assigned to you), then you dual-license it to publish it under the store
mik: i'm of the opinion that we should have a legal entity that owns all the Scuttlebutt copyright
mik: barring acceptance by the rest of the community, i'm keen to have all Sunrise Choir copyright owned by some entity
- keen for us to strongly defend the value we create
mik: basically
- keen to license under AGPL-3.0
- also keen to have flexibility for future open source licenses and one-off proprietary licenses

merk(le|el) stuff

alj: my plan was to let old feeds rot (but still supported)
mik: keen to migrate old feeds, don't want to give up the cypherlinked library of knowlege we've accumulated
alj + mik: we still haven't figured out how to migrate data without breaking cypherlinks
- stupidly hard
mik: i've been talking with dom about this since the beginning
- latest convo: if we were to upgrade hash function, we'd publish a new message (say a message of type "upgrade:hash") that re-hashes the old message, then in the future when you reference the old message you use both hash ids
alj: this seems like a protocol thing, not an application-level thing
- messages are inherently application-level
alj: see what ipfs is doing
alj: with regards to using new feeds with old feeds, it can work if each server supports both, might be possible to use initial characters of old feeds to mark version, etc
mik: can we retroactively add the merkle-structure to old feeds
- can we maintain old hashes, signatures, etc for linking
alj: hyper-core is a fun idea, but afraid it's not seriously an option
- we have more meta-data than they do
- and use different hash and signature multiformats
- if we were tied to hyper-core, we wouldn't be able to add new protocol features that aren't already there
  - hash upgrades, signature upgrades, message dependencies, etc
alj: current view is to have replication meta-data (what you need for security guarantees) to carry around a payload
- then have a ssb specific payload inside (what you need for applications)
alj: post-call-edit: Nope, my current view is that there are of three levels of data:
- metadata relevant to verification
- Type and Option\<Timestamp\> as the interface shared by all message content
- actual content

@mikey6 years ago %HGEa2lf/9jtnPPuVolwmOdPzGSZH/xjpceUS21niFIk=.sha256

#sunrise-choir

sunrise choir retrospective #1

date: 30/10/2018
format: https://retrospectivewiki.org/index.php?title=Facebook_Reactions_Retrospective
attendees
- mikey
- aljoscha
- piet

checkins

observations

liked

mik: the Sunrise Choir is a thing meow!
mik: the train had no problem leaving the station and is building up steam
alj: not working alone on stuff
alj: high-bandwidth thought-sharing
alj: a few other ssb people seem to actually care about this stuff
- go peeps: keks, cryptix
- christian bundy (needs a medal for improving js codebase)
- moid, alanz
- dominic
pie: hitting some sweet milestones with napi + cross and stuff.
mik: that we share our notes, willing to be open
mik: seeing other people like our notes, dev diaries, etc
mik: our meetings start with a checkin
pie: did yoga yesterday and felt lots of emotions. Felt real nice afterwards.
mik: seeing the amount of effort being put in by piet and aljoscha
pie: being really disciplined with my timesheet.
mik: these regular meetings every week, good rhythm
alj: some problems with the spec getting promptly fixed in sbot
- non-canonical base64 (dominc made a module really fast)
alj: goofing around
alj: interaction with rust crate devs
pie: encouragement in dev diary thread from bob
- %wLK+6is...
alj: "lightweight" meetings
- unblock and/or give motivation

loved

pie: working beside Mikey once or twice. Arvo exercise was great.
pie: putting in consistent hours at the library. It means I feel happy and tired at the end of the day.
mik: seeing everyone's dev diary entries, seeing the progress that's happening
- yay active radio comms (not radio silence)
pie: writing rust
pie: learned a shit load in the last few weeks.
alj: freedom
- realized i could walk away from the computer and think for my work
pie: being able to call Mikey and checkin that he's happy with progress. And hear that the problem I'd hit happened to be a known issue was a sweet bonus.
mik: when we got to see aljoscha's face

made me laugh

pie: when my boss turned up to a picnic in a sweet dress. Best boss ever.
mik: when piet calls me his boss
mik: aljoscha being pedantic

surprised me

pie: prebuilding native modules is crazy complex.
alj: got numb regarding weird ssb behavior
alj: effects of regular dev diary writing
- somewhat stressful. felt like had to reach milestones. learned to be ok with not
- good to have to regularly communicate outside of the cave
- gotten easier with time, more okay with sharing whatever
alj: spec macro-organization is hard
alj: feel like I'm alone with my level of detail of protocol knowledge, which makes communication difficult
- maybe talk more about this later.
alj: ~~time~~ self-mangement
alj: time spend working vs time spend accomplishing something
alj: thinking about my output as being paid for, being able to calculate "How much did this cost mik?"
mik: aljoscha doesn't have a face in meetings.
pie: it's taking quite a push to bring the native bindings thing to a useful point.

made me sad

pie: felt like I couldn't keep up with all of aljosha's work. Want to be able to engage and support but haven't that well.
- don't want alj to feel alone fighting the demon
alj: my English...
alj: one month of work but still no "real" impact on the protocol
alj: so much programming for programming's sake (appeasing the language, fitting into traits) rather than inherent complexity
alj: quitting the tutoring job
alj: no single stable place to work
pie: I miss alj's face.
mik: haven't been consistent in doing the things i say i'd do
alj: (decimal representation of) floats
pie: doing compilation on my laptop is a bit of a drag.
alj: merkle trees not working out
- mik: why does it work for dat and not ssb?
- alj: ssb has a strong guarantee about past messages having existed in the past.
mik + pie: hearing that aljoscha is having a tough time outside of work
alj: having a tough time outside work =P
alj: being the only angry one

made me angry

alj: corner cases everywhere, absolutely everywhere
alj: floats
alj: Lack of thorough engangement (at first) from Dominic on small but still crucial details

discussion

being alone in the details

punt (i.e. delay) until next week, we can spend time focusing on this then.

in the meantime, let's keep a watch on it, mindful in case it gets worse.

actions

???

normal meeting stuff

bindings

js object equivalent of https://github.com/ssbrs/legacy-msg-data/blob/master/src/value.rs#L138
run this, then create a js object with the right keys (with the content type T being the js object from above)
bindings to https://github.com/ssbrs/ssb-legacy-msg/blob/master/src/verify.rs

action:

alj will write blurb about intended js api for rust module
pie will handle making js bindings, try native and fallback to wasm

@aljoscha6 years ago %d5rwEHOIsGRCJMJWYM+6mquFXfYWqQU4Mq2zoj2c+hQ=.sha256

#sunrise-choir

sunrise choir - 06/11/2018

attendees

piet
Aljoscha

agenda

checkins
serde stuff
- alj gave piet a sweet intro to serde!
is alj blocked?
- compact format is mostly done
- ongoing discussions about sigils
- could start working on new format
- started defining bpmux (also blocked by futures crate)
  - could implement in js now, rust later.
- feeds by type meta data thing.
- might look at binary multiserver(addresses)
libsodium / or ring

@mikey5 years ago %4BF8xlWJh/httfB+L5aElZIyyLFor2rpLslFHqtQgPc=.sha256

#sunrise-choir

Sunrise Choir - 13/11/2018

Attendees

Agenda

checkins
piet update
- current benchmarks
  - what does this mean for bindings?
mikey wants to find additional funding
- after midway point, hopefully have visible momentum
- keen to see if Dan and Zelf and SSB grant writers are keen to pitch on our behalf
- Mikey is personally not convinced that SSB is going to succeed in finding much more grant funding, we've been lucky
  - we've also been declined from more potential funding
- the Sunrise Choir was a way to be more attractive to funders: clear team and clear roadmap

Actions

Ask for expert help
- Dominic?
- Matt?
learn if our binding approach is not being optimized
- is our constructor monomorphic?
- Try to see if profiler can tell if it it's de-optimised

User has not chosen to be hosted publicly

@mikey5 years ago %icjVKwhcZPIdEmc9tVeWLi/HkP5DLEBkoinxtE0qjmY=.sha256

#sunrise-choir