Minimal sbot in C: Dev Diary

Applied to #ssbc-grants : %SQ6BGiW...
Accepted: %tKE8YsM....
Project repo: %4w0BO1K...

Implemented event loop and receiving secret-handshake connections: %FZDUM0W...

@C-Keen @C-Keen-Work
You can push to a branch of the repo, and optionally @-mention me. Or any other way you feel is appropriate

It can now respond to the whoami command: %DrWoXIQ...

Dev update. I left off on this a few weeks ago because of personal things and having trouble with the asynchronous IO, and picked it up a little a week ago: %RdaAmAm.... More recently I have been focusing on ssb-npm instead where I have been making progress. Today I did an easy thing for the C implementation which was to implement testing the server's secret-handshake implementation using shs1-testsuite: %MfSXBJs.... I am thinking next I might try to implement message verification and signing, which the server will need later, and other projects may be able to use as well.

Some progress, including debugging substream handling and implementing the blobs.get RPC method: %yW3j6xL... %yRRuyXO....
Implementing outgoing peer connections is underway.
I also implemented some separate tools to read flumedb files (flumecat, ssbget), and from those experiences I think I will try to use the same file formats in this project for storing messages and their index by id.

Implemented outgoing peer connections: %ski5T3O.... Tested with shs1-testsuite. Uses asynchronous DNS via GNU extension getaddrinfo_a, or synchronous getaddrinfo if make variable NO_GNU is set.

Started implementing blob gossip: %lXnxSNE.... Need to figure out what state exactly is needed for this.

Implemented blob gossip: %07LR0NA.... There is a bug where it the server can get into a send/receive loop, which I am debugging now.

Fixed the send/receive loop bug [%Gid8YrX...].

%I1liEhf...: Implemented the get RPC method, to read a SSB message from the flumelog and keys hash table as written by scuttlebot, reusing code from ssbget.
Also implemented accepting connections via local unix noauth socket.
Also added a manifest method [%PIqXk7G...].

Some of the RPC methods I've implemented are not strictly required for replication, but they help with testing, debugging and administration/usage, and mostly don't add a lot of complexity. (i.e. blobs.{add,has,size,want}, manifest, stop). So I think this is okay. I might change my mind though if I can figure out a good way to test blobs gossip without blobs.want, for example.

The filesystem operations are synchronous/blocking. That is reading, writing and statting blobs, and reading (and eventually writing) messages and message indexes. It would be nice to make these async, to ensure responsiveness and minimize peers disconnections due to timeouts. This could be done using an additional thread(s), or aio - but it would increase implementation complexity. For now I want to focus on feature completeness and correctness.

Next things to do:

• fix an issue with blobs.want
• implement ooo message gossip
• update keys hashtable
• message verification

Later things to do:

• message indexing by feed+sequence and latest per feed
• createHistoryStream
• ebt
• contact graph indexing and traversal; requesting remote feeds
• local peer discovery
• gossip scheduler; pub message indexing
• more testing

More things to do later, maybe:

• message canonicalizing and signing
• out-of-process plugins
• index backlinks
• web[socket] server
• hsdt

Status: blobs replication is buggy and needs reworking. It is harder than I thought. Estimated work done: 20 days, 10 remaining. Motivation low.

I had a call a few weeks ago with @w about my status on project (re: %ecWj6gQ...). In the 1-2 weeks left of the grant time I didn't feel I would be able to complete the goal of the project. I was encountering difficult to debug and reason about situations while trying to implement the blobs replication and I/O scheduling.
However, there are still some things I could try to do:

• Implementing message signing and/or canonicalization, and verification, could be done orthogonal of I/O, and could be useful later. But I don't much of a use case for this.
• Implement ssb-tunnel. An sbot not even implementing SSB replication but just the ssb-tunnel API, would be useful.
• Implement client and server commands just for secret-handshake. This could be useful for building a sbot in a modular way, to have secret-handshake factored out into a separate process. It could also enable wrapping other protocols in secret-handshake.