it's exciting to see the go and rust implementations coming together :D

@keks status of muxrpc. v7 (which turned out to not be quite ready yet) didn't change the wire format, just introduced a "control stream" concept, which tells the other streams when they should be paused.

I meant to ask this back than but time is passing quite fast for me in the last months...

do I have to use the control Chan as a client or does the server just throttle me? How does it work for v6 peers?

@cryptix @keks yes, the design is if the control channel errors it will fallback to backpressureless (v6 behaviour), so it's backwards compatible.

@Maximilian you are quite correct, see push-mux readme

small-ish update to our progress so far: The refactor of go-muxrpc using the luigi interfaces @keks specified is looking great! We tagged v2 today as a first shot.

We got async, source, stream and duplex calls functional and also found some bugs using interop testing against node using muxrpc@6. I'm sure there are more corner cases we need to handle more careful but this already feels great! @keks is currently working on convienence tools to make server handler and client caller stubs from a manifest. Previously we only had a semi-solid interface to make client calls but the server handler api was a mess.

Once we had some good interop test coverage, I couldn't wait and made a dystfunctional but verbose test sbot that just logs incoming method calls with arguments. It was quite educational to see the call history from different clients. I tested git-ssb, patchfoo and patchbay so far. Check out gossipgopher-log.txt for method names and their arguments.

I'm quite eager now to implement the blobs interface (has, createWants, etc) and gossip.ping since they are small but very usefull. A bigger roadblack we need to tackle soon is the message verification code. Once we are passed that we can plug margaret in and start syncing feeds! By then we should also talk about plugins and indicies... cc @dominic @Aljoscha

• go-muxrpc: %8L9AKN2...
• gossipgopher on github (also on %D/fT3SG... but seems like blobs are missing.. :S)

ps[before I losse the links again]: There was a lot of discussion about the msg canonicalization but I think this github issue has the most content. There also is andyleaps msg sign and verify code for past reference but IIRC it somehow choked on unicode characters in messages..

@austin : you can find some history here: https://github.com/ssbc/packet-stream/blob/master/README.md

Oh, also: I think it is about time for minimal sbot.

merging %dHQJeiQ...:

hey @moid!

We have two Go implementations, one of which halted over issues with json formatting?

I assume you mean andyleap/go-ssb and the new funded #go-ssb efforts?

I fear the protocol guide is glancing over assumptions the nodejs implementation inherreted from v8. Having to strip the signature from the message object while keeping the insert order intact will be a struggle I'm not sure yet how to overcome.

Having tried andyleaps implementation again I also feer problems with utf16 encoding but I can't pin-point it yet.

yup! I'm also pretty happy on the progress we are making now. small things to add:

new github org

we moved all the related code to github.com/cryptoscope after considering @dominics comment %IV6/jqs.... We are happy to move any or all of them to to ssbc but didn't really know where to draw the line between pull-stream-ish libraries and the actual ssb code.

We kept the #golang vanity import paths (now go.cryptoscope.go/muxrpc for instance) to later change the location of the repos but I felt an urge to have CI and an approachable issue tracker (i.e. you don't have to be (close) friends with me for me to see it).

the gossip code doesn't have any friend graphing yet

while our go sbot can now talk to other peers and sync, it doesn't care yet to whom it talks. I'm very much with @mix re %z28ZLyE.... I want the friend distance check to be considerd when peers are dialed and also respectful about which feeds they sync. I'll closly study %FOyO23x... for this.

apart from that the code is still messy and buggy, it has overhead in network and disk-usage and also might deadlock connections. I only use it to talk to other peers on my local network, I really don't advertise you use it to replace your pub yet.

hey @keks, thanks for looping me in. Where are you keeping the flumeview-multilog code ?

hey @kik!

Do the go-muxrpc and secretstream go libs mean we can communicate with a running "official" sbot (the one written in js)?

yes exactly! the cmd/go-sbot is our attempt at sbot server with it's own db and gossiping. The client is a bit burried in all the messy boilerplate we are cooking currently.

If you want to experiemnt with a simple client calling js sbots, i just pushed an sbotcli branch as an example of what is required. It can call async commands, for example sbotcli call whoami or sbotcli call status. And there is another example of draining a createHistoryStream source. See sbotcli hist --help.

repo links:

• git-ssb: ssb://%PgnHkktlss1qI/MW26FlgsrLXUWD/5WDN0uCTqImj38=.sha256
• sbotcli: https://github.com/cryptoscope/sbot/tree/sbotcli

cc #golang

sorry @mix I think the flumeview like is meant literally. The code is all Go but you can find it here: https://godoc.org/go.cryptoscope.co/margaret/multilog

This multilog will do the book-keeping of which sequences in the single received log are belonging to an individual users feed but its not integrated yet in our gossiping.

Can't wait to have it in though and add query helpers to query all feeds for type:contact.

hey @cryptix that's a dead link for me D:
Don't completely understand the use case yet but look fwd to reading the docs

1. Do you guys consider you have exhausted your ~1 full-time month of development? (You may have decided to stretch it on a longer period)

keks: Intuitiveley I'd say we did more hours, but that's hard to say because I didn't track my time. We are working on this part-time since the grant was accepted, ie. about half a year. While at first I could concentrate on this project fairly well, I later had other obligations which led to me switching context a lot, which has somewhat hindered my productivity every now and then.

cryptix: Keks did a lot of work in advance (implemented luigi and margaret on it's own) before I had the chance to collaborate because I was still tied up in other projects. In the beginning, each of us working on their parts was okay. Towards the end it would have been better if we had more time together to figure out needed changes to the design we started with. In total, yes. I used more time than I allocated in advance. I planned to spent more time on it afterwads in any case but setting clearer boundaries might have reduced unfinished stress.

2. What are the things you have achieved that others can build upon now?

We believe what others can build upon the most is

• An working foundation you can extend upon to support more commands. Namely the ability to define muxrpc handlers for the different parts of the system.
• working message verification
• the new muxrpc interface with luigi streams should allow client development in Go.
• another implementation can be a helpful reference for future implementation efforts

3. What part(s) of the initial proposal still need to be completed? (No need to feel shame if not everything was done, this is a learning process not an exam...)

For a while now we have all the individual parts done, but we still need to tie them together in a maintainable way. This is a bigger task than we anticipated, and we've been stuck doing that for roughly two months now. We're still working on that, though :)

There are a lot of small things that need to be done which we set out do but were pushed back because the integration of the big parts didn't go as smoothly as we naivly thought. Firstly, we need to be able to craft/publish (format, sign and replicate) our own messages in pure Go. Also, blobs support is still not implemented, but that is a low-hanging fruit as well.

The testing setup on the other hand is a wild mess of hotglue and could use a cleaner setup. Right now I (cryptix) craft a couple of messages in a nodejs instance, manually by hand. Sync them by running gossip.connect and then check in the new go-ssb repo (repo is our name for the .ssb folder with the database) as testdata for our unit tests. Now that we moved to multilog, I need to redo this but don't want to redo this every time we change something. Personally I prefer to start with message crafting as that would scratch other itches as well but feel like it opens another can of worms.

Another thing we had in our proposal was the development of a plugin system. Unfortunately, this won't make it in the final release, at least not in it's full and more interesting form. However, we started a community process to design a sane plugin system that solves everyone's needs. This seems like the better path to us, because there are many people who wrote much more ssb code than us, and it would be ignorant not to rely on their experience.

To summarize, we are really not sure what the next best step is in our current state.

4. What did you learn about the project that can help others do similar initiatives in the future?

Also we learned it's really hard to estimate how long you're gonna need to buid something! In our case the perceived last 20% of the work take more than 50% of the time, because everything needs to play well with everything else and we wanted to have as least hacks in there as possible.

I think the most unintuitive thing in implementing an ssb client or server is the message verification and hashing code. We documented these in some threads, which we hope will be read by others who plan to do something like this.

4.1 Were there any part that was more complicated than initially planned?

Definitely the verification (see %M/qtHwv... ), Hashing (see %SJR3ZAB...) and time estimation.

4.2 Do you have any recommendations for the community about technical choices?

We think SSB has grown and evolved a lot in a very short timeframe. We consider this a success. At the same time, we start to see the problems with this approach: The message encoding before signing and hashing is a mess, and JSON seems to be not a very good choice for encoding messages for storing and transmission in the SSB case. Also the sets of assumptions different implementations in different languages bring to the table vary widely. At the same time more and more people start relying on SSB for social media and productivity.
We think we should slowly move our stability-velocity trade-off a bit towards stability. This allows better collaboration between different sets of users, and hopefully fewer breakages or unexpected behavious.

We consider Go a good language for this project, as it's very different from the reference implementation. The strong typing forces attention to details and the absence of the npm ecosystem, especially all the missing ssb modules, put you back into square one which gives you a chance to reconsider assumptions. That being said, it might have been a good idea to not hack on our own database implementation from the beginning, but to use an existing one until we have something that works.

5. What did you learn about yourself that may help you be more effective in the future?

keks: I've noticed and read about this before, but with this project I really had ups and downs in my motivation to work on it. Currently I'm concurrently working one (soon two) other jobs and I've started working on my master's thesis. Sometimes I feel like the context switching introduces churn that inhibits my productivity, other times I have the feeling that it's helpful to just switch the task to something else that needs to be done when I get bored of one task. I guess it's a trade-off and I haven't found my sweet spot yet. Probably also depends on the task and my general state of mind. Another thing I knew about myself before is that I have the tendency to generalize and refactor stuff very early. My motivation is that we'll have to move less code to the new form, but that comes at the cost of maybe doing that more often than necessary.

cryptix: I learned that I have to improve my communication with others. I have a tendency to swallow problems up and try to solve them on my own. Adding to what keks said above, I have experience with this kind of multi-project churn. I'm honestly not sure if there is a sweet-spot but it is very personal. In the end of the grant time I sometimes had the feeling that I shouldn't bother keks because he already spent his time and it was now up to me to get stuff working.

6. What did you learn about the community by developing within it?

Due to the nature of our project (highly technical and experimental), we haven't had much community interaction so we can't say much here. We didn't feel that much interest but explained this the far out nature of what we set out to do.

There were two topics we discussed with the community: the message encoding and the ideas on the plugin system. Talking about these was great, and even though we do not really agree on a good portion of questions, I think everyone can understand and sympathize with everyone else's viewpoint. I'm confident we'll work out more agreement in the future.