Tightbeam - Week 2

This week went fast and much happened in terms of Tightbeam development. If you're just here for the new feature list, scroll to the bottom. for the novel, continue reading.

This week pretty much started with the continuation of my REST api -- a small ssb-plugin -- that could be used to query messages, likes and replies from a running ssb-server. It was super basic, but the POC got the job done. You can find the (now old) backend service here. If you ever feel like building something simple, take the code ;)

I managed to get markdown working too in the app itself, which was pretty nice!

And then Thursday happened.

After @mix had put out his Patchbay patchqll branch I had to try. It went south fast with new features and functionality exploding into the app.

All of a sudden I had access to pagination, threads, private messages, the whole shebang! I pretty much spend the next few days after that integrating graphQL queries and data until I was forced into some refactoring on Saturday.
It was then I realised that I needed some extra stuff. Stuff like being able to pull images from the server and making actual posts. So I left Patchbay behind and started working on an "improved POC" of "Tightbeam Hub". This hub would need to do 3 things to begin with:

• Host one or more SSB servers with PatchQL for the posts & a REST api for the rest.
• Be able to proxy incoming signals to the right ssb-server (added bonus: token based "authentication").
• Combine the ports of the Graphql end-point and the REST end-points into a single HTTPS signal for convenience.

on the other side, the hub would need to be

• Inneficient enough to not advertise big ammounts of people all "creating accounts" on a single provider, we're DWEB here.

So the "hub" sketch ended up to be something like this:

I put the code for the new hub on my gitlab server. It works pretty decently, but it obviously needs a better (JWT-based) authentication system as tokens are pretty unsafe. It's also just a POC. If you roll this anywhere, make sure your firewall is up, when calling for the service from a distance, setup HTTPS (instructions not included in the readme of the repo).

So that wraps this week up.

On the list of things that now started working:

On the hub

• Fetching posts, threads, likes, private messages, private threads and profile information through graphQL on POST http://localhost:4000/graphQL
• Creating new Messages (posts, votes, abouts, private posts, etc.) through a REST end-point on POST http://localhost:4000/posts/

On the phone app

• Start page with latest markdown parsed posts, with dates, likes, etc. paginating to infinity (or the end of your database).
• Thread views upon clicking on a post.
• Private message list with pagination ofc.
• Threads upon clicking on a private message.
• Working web-links (internal links, references and pictures aren't working yet).
• Ability to set the server-address of your hub as well as pasting in a token.

DEMO VIDEO

Other random videos created earlier this week:

• first patchql shenanigans
• thread are rad (IOS baby!)
• Music + code + vim = life (random older video)

Todo list for the coming week.

• Get blobs working.
• Fiddle around with the settings to allow continuous functioning of the app after changing server and token (need to restart now).
• Start using the already integrated feature on the hub side to post messages and go from a read-only app to a full blown app.
• fetch threads when clicking on post references in the markdown content of posts.
• Continue to research good practices and strategies to keep data-sets on the device for offline access.
• Cool things I didn't account for yet that will somehow be awesome and make me code all night.

Longer-term

• Profile pages & the ability to change one's profile information.
• Integrate features from desktop clients into the app (Who doesn't want to play chess "the nerd way" on the bus?).

Thanking some people

• The people behind #sunrisechoir. You folks are rock(et)stars!
• @piet and @mixmix for the amazing graphql stuff, code examples and pointers!

What are the implications of hosted key's? Shouldn't the keys be stored in the client where the log signing happens and then just appended to the log on the server, light client style?

This isn't to say what you've got isn't cool, it's great.

This is coming along well @Hendrik Peter. FYI the "where are the keys?" question is a really hot / dangerous topic. At the moment from your most recent diagram (and my guessing), you have all your keys in the hub.. because otherwise how is that pub serving private messages to you ... it wouldn't know which ones are for you without your private key.

What I mean by "hot / dangerous topic" is that some people will get very upset if you build a system which holds private keys on an external server. The problem is that this is a thin end of the centralisation + control wedge, and could lead to some massive security problems. This is something Rabble and verse have been through. It's also a hard problem because you want to be able to do private messages in a light way but doing it with a light client is VERY HARD... the only way with the way private messages are currently written is if you include the hub in the PM, or if you send every message to the phone and ask it if it's a private message for it , then maybe the hub could store a list of keys which it knows are you PMs, without storing the content.... this could be made lighter by not checking privates messages you publish, and immediately telling the hub it's a message "for you"... and only checking private message of people you follow to see if they're "for you" ... hmmm hard.

Anyway. I hope that comes across ok. I'm not upset at all, I'm just trying to be very clear that this is dangerous territory because I want you to have a good time and succeed.

precisely @Rabble, @Piet and @mixmix.

When I did experiments with the existing ssb-server I noticed that all content going in to the database (so not only my posts, but pretty much everything that gets stored) has to go through signing using the keys. I outlined that in my previous weekly. This essentially means that I have to download a post to the phone (or sniff it up from wifi), request the previous data-set from the server-side (since we hash new records from the previous record in storage), sign it on the client side, then send the new record back. at which point 1 piece of gossip caused 2 pieces go over the wire for a a total number of 3 times.

I learned a lot of new things last week though. Between hosting a pub, dissecting ssb-server, ssb-master and pretty much every module I came across in detail. (Without too much confidence) there seem to be 3 paths that I could take from here with the private key concerns:

1. Lowest hanging fruit: Keep it as it is and limit the backend that I wrote this week to only being able to deal with 3 or so ssb-servers. This is actually already the case as spawning more ssb-servers in the contraption I made is super inefficient in terms of storage, CPU and Memory use. With this the product lives as intended: you have small hubs (NAS machines or private VPS boxes) serving 3 or so accounts in a house-hold, but yes they hold the keys.

2. beefier effort: Follow in the tracks of @andrestaltz and get ssb-server working natively on IOS and android. Serving a built-in ssb-server and bridging external non-java/swift/Objective C & JS into working IOS and Android modules. Then offloading old blobs (which after dissecting much of what I've seen so far is the biggest storage-eater) to a connected backend. My focus here is on IOS, I want to get that working; Users can already choose an android app with Manyverse out there. Apple is however incredibly picky in their reviews, so concessions will be made.

3. Over the top Rocket science: Rewrite ssb-server and it's modules to go for a more softer approach in signing, But doing that will effectively make private keys worthless, so that's not an option. or dive deeper into how messages are signed. does hashing messages really require the private keys? and could I instead sign my message on the phone, then have the server-side backend add the hash to link it into the chain?

Since I have Andre's attention here (sorry that I keep tagging you into things).
I think integrating patchql into mobile apps (doing things client side) is a bad idea, unless a full sunrise choir backend is included. It's super inefficient to run an ssb-server, have a patchql server make requests to that ssb-server for content, caching that and then serving it forward. Apart from the amount of moving parts (and what could go wrong), there is the phone-battery, I already see my battery-indicator tick away like the seconds notation on a digital clock when having Manyverse open in its current state. I don't think that will improve with patchql joining the mix, unless it replaces existing moving parts.

I think integrating patchql into mobile apps is a bad idea, unless a full sunrise choir backend is included. It’s super inefficient to run an ssb-server, have a patchql server make requests to that ssb-server for content, caching that and then serving it forward. Apart from the amount of moving parts (and what could go wrong), there is the phone-battery, I already see my battery-indicator tick away like the seconds notation on a digital clock when having Manyverse open in its current state. I don’t think that will improve with patchql joining the mix, unless it replaces existing moving parts.

@Hendrik Peter The current indexing workload that the JS SSB stack does is very intense, also on desktop, but there are strategies to decrease that workload, some of which are being implemented into patchql already. Also in Manyverse itself the indexing workload has reduced with the adoption of wasm libsodium and other things, so that when I ran a test on a modern smartphone (Google Pixel 3 if I recall correctly) the onboarding and indexing took just 40 seconds. Initial onboarding also depends on hops count and/or how many accounts you follow in the beginning. There are a lot of ways to improve the onboarding CPU workload, from "doing less work" to "doing that work more spread out over time" to "using faster engines", so that I wouldn't dismiss client-side processing yet.

Doing things server side with lite client really worries me, because it's a return to the current state of the internet, which is heavily dependent on servers. It's not much better than ActivityPub. I think SSB's big appeal is that it's a bold return to the end-to-end principle and works perfectly off-grid, from computer to computer in any circumstances, which also has socioeconomical benefits for underconnected communities.

But I'm loving what I see so far with Tightbeam and it's refreshing to see new apps flourishing. : )

Ideas:

1. hub included in PMs

the intermediary solution "my hub gets included in every private message I send" would reduce the load on phone being forced to decrypt, and create known vulnerability for reading messages BUT mean I could not have my key stolen

2. a new sort of private message

• leak to: meta-data to reduce the load on account that you know are on phones
• keep text body encrypted
• e.g.

  // msg.value.content
  { 
  type: 'envelope',
  to: [ '@+UMKhpbzXAII+2/7ZlsgkJwIsxdfeFi36Z5Rk1gCfY0=.ed25519' ], // @andre-mobile
  text: 'asdasdlhxinsdxjkajalkdxkjahdsx' // private message for andre-mobile + others?
  }
  

• the to: field is only for recps who have declared they are phones. This means that here only andre's phone knows to bother trying to decrypt. Other desktop recps might also try it out and could have their meta-data as recps preserved.
• you could obfuscate with dummy to: recps (like pick some other random users)
• when andre replies to mix-desktop, he can use classic private messages, so not everyone is taking a privacy hit. He knows to decrypt his own messages ...

the more I think about (2) the better this idea seems ... it's so simple and not so terrible

The message type is envelope because you can see who's it's too but the contents are hidden...

@gwil @ home

What if ssb-patchql could be provided these in a header of a HTTPS request from the client, store them in the request’s context, and use that to return the appropriate results when resolving the query?

That introduces 2 small problems:

1. The wire: It's much safer to send temporary tokens through the air. If tokens leak, they are going to be valid for a limited time anyway so damage is containable. They are used to authorize you agasinst something, If you send the private key over the network and it leaks you're in an entirely different can of worms, the private key is in some sense "you".

2. The database problem. The location of the private key defines the place where all posts are signed and hashed to the database (correct me if I'm wrong), so if the key is on the phone, the phone needs to gather new posts.

I'm going to research ssb-gossip and the inner-workings of ssb-server a bit better to understand it more and see if I can divide tasks between devices. Alternatively I just go to the approach of doing everything client side and using the server-side as a blobs-dump.

@Piet , thanks I'll read through that, once again you present me with really tangible stuff!

as a foot-note: I'm flexible enough really to go into many directions from here and there's no clock ticking either I wanna do this "right". Right doesn't mean that everyone is happy all the time though, silver bullets don't exist.

lite client works fine for public messages... but private messages doesn't work so well.

Adding metadata back to the private message is bad. It's worse than a centralized system, because at least only the center knows who you are talking to. In a decentralized system without metadata privacy, everyone knows who you are talking to.

How about: download all private messages from your direct friends. discard messages you weren't able to decrypt. friends of friends arn't able to send you a direct message, that's okay. or if a friend is also in the thread and they reply, you could pull down those messages.

But actually, I think we can actually get the full peer good enough to run well on a phone, and this is by far the most worthwhile goal.

I’m going to research ssb-gossip and the inner-workings of ssb-server a bit better to understand it more and see if I can divide tasks between devices. Alternatively I just go to the approach of doing everything client side and using the server-side as a blobs-dump.

Is starting to sound good ;)