You are reading content from Scuttlebutt
@andrestaltz %b6nlgiAu3ZWkLqKnvkU1T/9PZCfiqSU/Ujg1xRmD/64=.sha256

Dev diary: building an SSB app for Android

Started making an SSB app for mobile.

Using React Native. Probably will eventually use Cycle.js with TypeScript and React Native, but for now the most important is getting a proof of concept working.

Started the RN project, and then simply did npm install scuttlebot. Of course everything broke. But it helped me to know what broke.

Installed rn-nodeify which gives me some basic node packages like fs, dgram, path.

Then noticed there is no react-native-sodium library. There is gladly sodium for Android. So next step is developing the react-native-sodium library and then from the RN project settings I can alias chloride as react-native-sodium everywhere. That's what keeping me busy now. Main challenge is simply finding what is the full API for node's sodium so I can mirror it correctly. I would be happier if @types/sodium existed, which doesn't.

@andrestaltz %xGXFLQHFXleuUwpqlF5UgxTpCTlrk+T5UgQ88nMV8V4=.sha256

Found chloride-test, thanks Dominic.

@andrestaltz %q9huK9p6C1+9/DcI0pr+9l+f0ixwGr/AZdmUp3P91dk=.sha256

I might be already utterly screwed, because RN native modules can only be asynchronous, and it seems like sodium has mostly synchronous APIs?

@andrestaltz %Pri3co1zwn7GhJaQZhCxmNLWVLFDA9OGywDJMElSIaE=.sha256

Oh, scrap that.

libsodium-wrappers which is libsodium compiled to javascript via emscripten if performance is important but code size isn't

libsodium-wrappers it is then. Just trying to make that work.

@andrestaltz %qvjhv8qqqRAeFd4ibnllqAiK657I85muWVkyf1R7URc=.sha256

Ended up using sodium-browserify-tweetnacl which was the only one that seemed to load as a module in RN's packager. After hacking some packages in node_modules to refer them to use the alias "chloride"=>"sodium-browserify-tweetnacl", I'm now trying to get ssb-keys to work.

@andrestaltz %GOeicuH1nNAB7QFizuwr40IH5M0n54Y2h/b93MKaXEg=.sha256

device-2017-03-17-233002.png

Yes! I got ssb-keys working. It generates a key in this example. :D

@andrestaltz %W2tcyh3sIKhGeSwYk9pIKWLl4x1yyVPiUeCBOb4vnQ8=.sha256

Sorry for the huge image.

@andrestaltz %dloSUH3tWL24d8dtVq36G8ffyKsvP6ioLR5BI+Q7PxM=.sha256

Repo if someone wants to follow %+gCL8LS...

@Dominic %OmSEZu2MsIOdYgl+iF9SPskDpLOuM8NB1nffb4GaGX4=.sha256

@andrestaltz chloride/small gets you sodium-browserify-tweetnacl

@Dominic %RcS5SIVUEdcT0jJcwxuOcGH3QY7FsI/p0n4jR1IrUx4=.sha256

what sort of fs access do you get in react native? does the polyfill give you everything?

@Dominic %5s4WyBdM0sye2YtLIm80Yjd8j/37DxAmy+PcdoW2MyY=.sha256

also, what about net?

@andrestaltz %ze17/3Wi1AsmYHmTdiUSzCAkaLuZy/IjvgjKDAMy888=.sha256

net is covered already by https://www.npmjs.com/package/react-native-tcp.
fs is covered by https://www.npmjs.com/package/react-native-level-fs which in turn is covered by mafintosh's fs on leveldb, and leveldb itself is in RN's AsyncStorage. In the end, all goes to AsyncStorage which is just a cross-platform abstraction for "storage, but expect things to work asynchronously". http://facebook.github.io/react-native/releases/0.42/docs/asyncstorage.html It's backed by RocksDB, if available.

@mix %LWov7DV1umtuGwg91mHF/QF3Ar5AUZfDqwBoi4n9RbA=.sha256

@andrestaltz this is super exciting :sparkles:

I you get ssb servers running on mobile, this space could get very radically transformative

@mikey %Dd4ZrH7s+SVFJijnd41/cMP24WvpCmMEjgke4I/LY+o=.sha256

I might be already utterly screwed, because RN native modules can only be asynchronous, and it seems like sodium has mostly synchronous APIs?

having JS crypto is great for a proof-of-concept, but its at least an order of magnitude slower than native crypto. is there any way around this restriction, or any plans for React Native to support synchronous native APIs? and if not, are we willing to write and use only asynchronous APIs for sodium in order to have first-class mobile support via React Native? @Dominic

also hey roots working in React Native already: @gregkan @sararrrgh @iainkirkpatrick, it's happening.

@andrestaltz %ytJ4susH5xo43GLqS+MHaRjgnQkxJeTeOVe7Xp+/urw=.sha256

Yeah I thought about the same, @dinosaur. I'll see how far I get and how it feels. Our choices are: (1) reimplement SSB entirely in Andoid/Java, letting go of the existing JS code, (2) write an async-sodium library and reorganize our existing SSB code to use that API.

(1) is the long-term plan and we should probably do in the following years if SSB grows.
(2) is probably worth trying if (and only if) the JS crypto feels truly slow to users.

@mix.exe %Jy1IpCfBASkY+CdEVwFi0rZ9cvsQMDw4EGQSGOk30j8=.sha256

I estimate the JS crypto will just make private message threads too hard to run on mobile. That might be ok to start with

@mmckegg %V14+A5ayQJby24Vt3k7Sx0xSogZAflT326T4oRUpF+c=.sha256

@winmix hmmm, as long as the private messages are indexed, it shouldn't be too bad. I'm now indexing them in patchwork and the performance is much better.

@andrestaltz %QIxwNWIsVrqgkHT1sm6kREgf4UiHGhOYEo+GjpPu7Nk=.sha256

Note to self: if thick layers of abstraction for the db (react-native-level-fs to level-filesystem to asyncstorage-down to asyncstorage to RocksDB/SQLite) turn out to be too slow, I could try https://github.com/litl/android-leveldb, and in particular, making a library react-native-leveldb.

@andrestaltz %SuWARcLeMgjwPDfhWe8c/yYY8TpX41ufZJ1TUYvfRP8=.sha256

To be more precise, that library would be react-native-leveldown.

@andrestaltz %XWEBbbXSp1FfSMu4dDatWuo9TorMoJ1ouvxOjx3i/3A=.sha256

Milestone: got storage and fs to work so that ssbkeys.loadOrCreate works as expected.

@Dominic %dBTvd7fCWqhHpA9wYKThoa+xGEp3R9UD+B+Jb1GqzFs=.sha256

@andrestaltz sounds like good progress! it wouldn't be too hard to reimplement the crypto as async, because the changes would be relatively contained. The biggest difference is asymmetric operations, symmetric crypto is fast anyway.

The parts that might go slow are verifying messages (on receive) and decrypting messages (although, I have changes planned that will make the asymmetric part cacheable.

@Dominic %U37noIseSSPeTiYf73MDTTcZFNQu5CikQZwi35SmJaU=.sha256

I expect the storage abstraction lasagna be a problem.
It doesn't expose the right low-level primitives, but i did find proper native modules: https://github.com/johanneslumpe/react-native-fs

that could be adapted, though.

@andrestaltz %O1CeTMQd3X0dGqtJcaVky1t+b68j+kWLnPoXq1lqOYE=.sha256

I saw this react-native-fs, but noticed it doesn't mirror the node fs module (e.g. it's promise based). Might be possible to work around that though.

@andrestaltz %DSNAFAD9AueoLnuTfOIY4tYxiLy4BqjvSXpXvF+mRXM=.sha256

Today's progress comes in the form a video, through webtorrent https://ferrolho.github.io/magnet-player/#6a2699e4b192ef86423180a517ba173eb8955c53
(My VPS is seeding this quite slowly. Every few seconds it chokes, then it comes back. If you download it quickly, it's because I'm seeding from my computer)

@andrestaltz %vaf8z5v8WP0TXtGKD/riF6t4HvLSy40PJoItACigXxQ=.sha256

Alternative link https://instant.io/#6a2699e4b192ef86423180a517ba173eb8955c53

@mikey %x5FDCJb4e9sTgVOP49O0oCLz+pLNdxVJHKbpESIyJb0=.sha256

hey great update @andrestaltz! :sparkles:

i'll try and seed the video on my computer for some time to speed it up for any future watchers.

you mentioned saving and restoring gossip peers. just in case it's helpful, that's handled here by reading and writing ~/.ssb/gossip.json.

@andrestaltz %XmQS5ORCDoS6/lzhMoGy6RMvtYFLW7r0Ca3RDg3x5zg=.sha256

Oh, good! Thanks for the tip. Interesting that it's not in leveldb.

@andrestaltz %BqbRjkQq+FVdtJCIIfw3JLcdFMzb9me7HJwl5r0cVVs=.sha256

This issue is making my life very miserable right now https://github.com/nodejs/readable-stream/issues/260

@Dominic %7Mi1sRL71hzA67sjBF+N6DwqXvopw+RLi1YyPiSpJrA=.sha256

@andrestaltz where is that coming from? we have nearly got all the node streams out of ssb, there can only be one or two more. My guess is that this is from levelup?

@andrestaltz %PZKDIE11CeeT/xUQ02XgPpj/DVw64e1tyQcqUCtSVCc=.sha256

Yep, it's mainly from levelup, and more specifically level-filesystem and level-blobs. I got over the readable-stream complications, and got stuck with level-blobs in an infinite while loop https://github.com/mafintosh/level-blobs/blob/adb3411fedc611ea05ba1a3462eab53aa7de4324/index.js#L259-L284

It was basically getting a key like /.ssb/secretÿ00001a and incrementing it to /.ssb/secretÿ00001b and repeating that ad infinitum.

I'm wondering if I need to do a fundamental migration/change in how storage should work. I'm not sure what do android developers (or react native developers) use for storing images, but I'd guess it doesn't need to go through React Native's asyncstorage which is like a localstorage of sorts.

@andrestaltz %RcCIQNMXTCllm8AMEuyEbIhbadUIUv4wQJ3G4iv/a2A=.sha256

And just to clarify: level-filesystem uses level-blobs to store every type of file content. These aren't just "blobs" as we know them in SSB.

@andrestaltz %fi1zAPpE1Zusvug+Flv4aiNelu94vdgrWijuljLd1+M=.sha256

Note to self: experiment building a library react-native-fs which maps fs API to React Native's AsyncStorage. This could replace react-native-level-fs and the entire level-filesystem jungle. Then, for ssb-blobs, I can make a react-native-ssb-blobs which uses android and iOS's own image storage thingy. Thank the scuttlebutt gods for making the plugin system in scuttlebot.

That concludes today for me, I'll sleep over this and have a fresh start tomorrow.

@Dominic %y72fKOFezGumRh+woxW86tXI1W5iRtN86PSopj1vGd8=.sha256

really, blobs should just be stored as files. leveldb isn't designed to store things like that, and will do a lot of extra work to do so. So if we multiblob work on one of those react-native fs modules we'd be good there.

@aljoscha %KU++TvaOHt/c0NceU5iHJagbyI25ifYZWT42eaqJCb4=.sha256

If a portability rewrite in Java ever becomes a serious possibility, we (you? everyone?) should also consider C. Everything either runs C or has an FFI to it. It might come with a lot of ugliness, but it for portability it is probably still the best choice.

I'm not saying whether a rewrite is a good idea at all though, just wanted to put this here...

@andrestaltz %+YZkftduXt8Ur937b+0AFTzhg6WxqarUfocxjAYjmiU=.sha256

That's a good point, Android indeed has an NDK (native dev kit) and I would say that is the long-term plan. Right now I'm in proof of concept mode, which is ultra short-term.

User has not chosen to be hosted publicly
@Dominic %rkfjFa8CmOg3wc/0sVX6JH57BKcljFOiimUVCQCErAo=.sha256

@xstt I think that Decent is based on the liteclient (relies on pubs, but stores the private key on the device), but @andrestaltz is working on a full peer (that is, fully decentralized and stores the messages and blobs on the device)

@ev %3OPJi0N9LKdL+d7G171GgzKkPN/pr4epGAXiuH4+Kng=.sha256

Yah, the lite client is pretty useless if your device is offline. However, not syncing the entire database might be a benefit if the device has metered data.

User has not chosen to be hosted publicly
@Dominic %aXs1k9gXy33twkJZyO7gLX9f9fBNosufCHlHxFu10+U=.sha256

@andrestaltz intends to write a new UI, but once he gets sbot working on mobile you could certainly combine it with a different UI, this one or another.

@andrestaltz %H0aQnj0qnMjnlQBZC10su1/Duz/Y9sqUgsT8VU3WXGA=.sha256

For now, I'm trying to make this a proof of concept. A full peer on the phone is important for LAN gossip, and making gossip spread like wildfire, specially for people who's only computing device is a phone. Down the road, if previous steps were successful, we could consider local db pruning (of blobs e.g.), hybrid client (syncs data only from certain handful of accounts, the rest works as a lite client), and whatnot.

@andrestaltz %FlnQdERRBopY42dnWHXsteOq0poQ7h2wNZHgRrzIclY=.sha256

Little bit report on (un)progress: I can't get TCP sockets to work, even though they really should, there's even a package for that https://github.com/PeelTechnologies/react-native-tcp

@andrestaltz %45pvfbbLyuEorN+uRql5aO1CTflaqg/947pvR/ngSxc=.sha256

Discovery: react-native-tcp doesn't support server.listen(1234), only supports server.listen(opts), while multiserver net plugin uses server.listen(port). I'll make a PR to react-native-tcp.

https://github.com/PeelTechnologies/react-native-tcp/blob/75afd35a6b2c1ce13dea74da236d75b6bfdcff0c/TcpServer.js#L66
https://github.com/ssbc/multiserver/blob/58073dc0fc7bed93460b3fa718d3114dc2d4ec3e/plugins/net.js#L21

@andrestaltz %CmzOARulxIFVfMPQJIKe7nf+sfGi9gUTtGZ1BShPtLQ=.sha256

No, I'm wrong, react-native-tcp creates the opt object if a number was provided.

@andrestaltz %gwuG1AUF0MJ/lrHglw8F7POM2pqzDlGyH2AfBstFK6s=.sha256

Yesterday Dominic and I did some pair programming to troubleshoot the bug. We ended up finding two bugs, one in secret-handshake/protocol, the other in tweetnacl. None of these were "our bug" though, so we didn't solve the problem. There are two vectors now: (1) try to reproduce the bug by making a smaller project with React Native, react-native-tcp and pull streams, to see if the interop between these parts has a problem, (2) use websockets instead of TCP. Web Sockets are supported in React Native even without a polyfill library to separately install.

@cryptix %Ml6UtQixcL4HxB/Etoazu97CTP5sQ3DtjPgHBKfGiZk=.sha256

Nice to hear that you unearthed the problem at least partially. I find these things extermly tiersome.. kudos to you for struggling!

I guess dominic already let you know this but ssb-ws is already a thing for the patchbay/liteclient experiments.. So not all is lost if (1) doesn't work out, which I'd encourage you to try since it would be less wasteful, I think.

@andrestaltz %vnXaemBwcXnaEkN5sRDB9Oww89rThCuSy2cg4Go1r6M=.sha256

Thanks for the pointer! (I was looking right now how to actually use WS)
These things are tiresome (I've been at it for weeks) but someone has to do it, and it seems theoretically possible.

User has not chosen to be hosted publicly
@andrestaltz %DNkPq77Gkv0ngNxvNYTAu88JB6Ar8auVoM3XIsd/t+0=.sha256

Did good progress today to solve a bug. Discovered that our tcp socket polyfill supports only net.createServer(onConnectionFn) while multiserver/net attempts to call net.createServer(opts, onConnectionFn)

Still a good while to go before I can get a functional sbot on the phone, but at least I was able to do some kind of replication of data between sbots, one on my computer, another in the phone. Still rough and can't take off reliably. 🛫

User has not chosen to be hosted publicly
@andrestaltz %zWkx/xnQ3lP6htUa7BtFOGUqWgVbNKrTR6QOOi5JJdk=.sha256

Update: somehow I'm getting sbot on android to talk to sbot on my laptop, but it's not reliable. I digged into the debugger and noticed that createLogStream wasn't sending out data from leveldb as it should, which led me to discover that...

asyncstorage-down@3.1.1 is depending on an old abstract-leveldown (v0.12.3) while that one has been updated to 2.6.1. So I'm going to poke around with asyncstorage-down and see if I can get it compatible with the latest abstract-leveldown.

User has not chosen to be hosted publicly
@andrestaltz %vxgOzJKkPgjiyzpSZSVdQapheIb1Pyzb1cY/mWQv9c0=.sha256

It's a pretty dirty job I wish I wouldn't have to do. :D

User has not chosen to be hosted publicly
@mix %4s33EDmShNxyaXsej1mw3Ej0pSc1vf/WNadeMmSBCL0=.sha256

@andrestaltz you're rad. I think it's important to note that you can't easily see how many people are cheering you on.

:fireworks: :dancers: :blush:

If there's anything we can do to support you in terms of morale, tech, testing, snacks etc let us know.

@Dominic %g8s9BmLfGkpMVYfZPi0X6f2Ru4q350PnIdLjKJ30rNg=.sha256

@andrestaltz we can make you some anzac cookies. It's actually a biscuit recipied designed for being sent by post to from nz to europe. They where a thing from WW1, but are now just part of NZ culture.

@andrestaltz %97lfr5pZoscncMWm3xESlCB9JzuzUZOGHD0zEpHfuS4=.sha256

Thanks people :D
No WW1 cookies needed. These messages are already enough support

User has not chosen to be hosted publicly
@andrestaltz %zgdJuymWT1GhlA5U1WK51NgszVs1Yi5Ly9bByMgBv04=.sha256

That said, I'm warming up to the idea of traveling to NZ and meeting all these geniuses. And the nature.

User has not chosen to be hosted publicly
@mix %KVjfZ+7NAh5AmTpGEryeohrNDeexIpDLSrk5mJixFlQ=.sha256

If you're gonna come, you should come soon - it's cooling off down here !
You're most welcome to come visit, would love to host you and show you around. @widdershin and @raquelxmoss are around here somewhere too :P

User has not chosen to be hosted publicly
@andrestaltz %HCY6YeQ3yHI+s7PWbD9WXkJL9rnUBjj7ZeaYiTbbj0A=.sha256

I realized https://github.com/Level/leveldown-mobile would probably help a lot to get speed improvements (compared to all the deep layers of abstraction we have now for storage). If anyone wants to help, porting that project to React Native would be good in the long-term. For now I can go with the inefficient solution as long as it's the easiest/quickest to get things working.

@Dominic %KhhYwvDPCdFqIniQnC5YBPaKGTNrnjwTQz9ZE0ICk1A=.sha256

And I was getting excited to see if I could even make anzac cookies on in my boat galley (well equiped for cooking, but poorly for baking)

@andrestaltz %6lndYneuKsH4IumtVr+uQwDPejvkozuMHLxEAegtA+0=.sha256

Update.

I've been struggling and doing slow progress mostly due to the feedback cycle being quite slow, I have to nuke node_modules and restart the app often.

I rewrote asyncstorage-down to be compatible with the latest abstract-leveldown, got all tests passing. But hit some problems because some db keys were intarrays and the way gt/lt/lte/gte ranges was implemented in asyncstorage-down had lexicographic comparison of keys, and I don't think that quite works with intarrays (it works well with keys). So I basically disabled gt/lt/lte/gte in asyncstorage-down, which made it not anymore pass all abstract-leveldown tests, but at least I got it working for sbot on android.

Next problem I hit, which I'm currently trying to solve is related to logDB. In secure-scuttlebutt v15 (not the flume one), we have this logic for createLogStream:

 db.createLogStream = Limit(Live(function (opts) {
    opts = stdopts(opts)
    var keys = opts.keys; delete opts.keys
    var values = opts.values; delete opts.values
    return pull(
      pl.old(logDB, stdopts(opts)),
      //lookup2(keys, values, 'timestamp')
      paramap(function (data, cb) {
        var key = data.value
        var seq = data.key
        db.get(key, function (err, value) {
          if (err) cb(err)
          else cb(null, msgFmt(keys, values, {key: key, value: value, timestamp: seq}))
        })
      })
    )
  }, function (opts) {
    return pl.live(db, stdopts(opts))
  }))

The important part is var key = data.value and db.get(key, .... I'm hitting the situation where data.value is actually an ssb message, like {author, content, sequence, timestamp} etc. It's an object. And down the stack below db.get(), bytewise.encode will fail to encode that object.

So currently I'm trying to figure out why does pl.old(logDB, stdopts(opts)) serve SSB messages under data.value. I believe data.value should be here some valid SSB ref.

@andrestaltz %BLYzWS2Re05wsd7bliIe8q/ukqQBQxQuMGNpCXrf+qs=.sha256

Edit:

-and I don't think that quite works with intarrays (it works well with keys)
+and I don't think that quite works with intarrays (it works well with STRINGS)
@Dominic %v6y5XYFhv+1Uf9BO5QJ2nRmNWuBAriBABJ/kSgdpczo=.sha256

sorry, my reply should be here: it's level-sublevel

@Dominic %UDswN5kEbYp0Q2aJLnaH2Z/MwUy7J0dWn/+Ev5eQnxQ=.sha256

@andrestaltz yes that will work! base64 is more compact, but unfortunately the lexiographic order of encoded base64 is not the same as the raw binary!

@Dominic %bm1iUDZfBKD4TWOU6G6009pXrmwDuFrqgiWgSYyWo3c=.sha256

@andrestaltz yup that will work!

@andrestaltz %K9d5Z/yZ65XM+Z2DD/2oM8fbbm3v+LBsOOk1mAbord8=.sha256

Good news! It "worked"! I got a message from the computer to replicate to the phone, and vice-versa. If it were an airplane, you could say it flew. But quite soon it crashed into the bushes. I'm getting different kinds of problems, it's very unstable (e.g. tweetnacl erroring on "no PRNG" even though it looks like it should be there). I'll keep on fixing these and eventually commit and push. The code is in such a crappy and patchy shape that I wouldn't dare pushing it. But soon, I promise.

@andrestaltz %j6Ym3Gb+/wK41nqZHhTpYPO5WoZY+lL2kv6zngWnzVM=.sha256

:raised_hands: :raised_hands: :raised_hands: :raised_hands: :raised_hands:

It works! GOSH this took long. I pushed code to https://github.com/staltz/poc-ssb-mobile. There are some crude instructions there on the readme too.

itworks.png

What works:

  • Basics: crypto identity generation, "leveldb" storage
  • LAN sync from remote sbot to mobile sbot
  • Posting feed messages on mobile sbot and viewing that on mobile sbot feed

What works unstable:

  • LAN sync from mobile sbot to remote sbot

What does not yet work:

  • blobs
  • ?
@mix %UcjE0gV01sHQFrqWxx0rE5l+nBcgweK99sdWk9a2fCk=.sha256

:boom: !!! :dancer: :cocktail:

@Dominic %T3tH0sOOk4dX9coIc4qE91xLqmq3hp2KOzr7ORk7C9k=.sha256

YUSS!!!
:trumpet: :confetti_ball: :dancer:

User has not chosen to be hosted publicly
@dangerousbeans %lSbC64l2oclPu9qsNKbUw1eWPXHaWjFb23kvspRI37g=.sha256

:sparkling_heart:

@andrestaltz %PD0ara9OKExZHijsgnvuDroyq34vVQr+7G0+Zt1X7TY=.sha256

LAN sync from mobile sbot to remote sbot

So this one actually works too. Both directions of syncing works. What doesn't work, which I'm investigating currently is: ssb on mobile should be able to fetch its stored feed and display it. But it can't, unless it connects to another peer. As soon as it is connected to another peer, subsequent calls to pull(ssb.createFeedStream(), something) will work as expected.

@andrestaltz %aewE7sq0qYYEDTpv4zfFCxXFl3axTqs4e4Hob+mDusE=.sha256

So two news:

  • I accidentally left Patchwork opened on the computer and my phone started syncing my real feed data!

long-feed-on-mobile.jpg

Was so fun to do this mistake and see things working :)

Second news:

  • Could not get two phones replicating their feeds in realtime through LAN

Because Google apparently pooped on Android: https://android.stackexchange.com/questions/49188/how-to-get-mdns-working-for-chrome-on-android

I figured I have to use something called jsDNS for Java and/or Android. Might be time for creating react-native-jmdns. Or I could try using react-native-ssdp and wrapping that as an sbot plugin.

@andrestaltz %R2OamkHwXsgMJiwj7eIhaxocp8vrLv1qHAyh/8LNnXQ=.sha256

s/jsDNS/jmDNS/

@andrestaltz %I5GrChJSKV4tNGfWW5hAd8X5SVOfRK9ljFaUAUjQZ/c=.sha256

Okay, found something that could revolutionise everything: http://www.janeasystems.com/blog/node-js-meets-ios/

Basically, running normal node.js on a different thread than React Native's JS thread. This would have the benefit of (1) not having to carefully hackily port node.js stuff to RN, (2) being on a different thread than JS for UI.

Point (2) was actually spinning in my mind already when I accidentally started syncing my phone with my computer's ssb feeds. It's not a good idea to have hard core SSB stack and UI in the same process or thread. Patch* already separates these, as far as I know. Even when you run sbot server on one terminal, and sbot feed, you also have multiprocessing going on. So it would be natural to expect multiprocessing/multithreading for mobile.

I'm going to give that approach a shot, while pausing this approach, but just for a while, I don't want to spend to much time on this detour.

@andrestaltz %1ZGNu5eqPYjk+b/lIk+MG7IspS9zNtsa9KKQS29L09I=.sha256

Another link (discoverable from the first, but not so obvious): https://github.com/janeasystems/node-mobile-react-demo

Peer-to-peer shared album app, based on Node.js on mobile running a PouchDB server, with a React Native interface.
The app uses Node.js to set up a UDP broadcast communication layer that enables synchronization of the PouchDB database between all the peers on the local network.
PouchDB uses MemDOWN for the levelDB adapter, which stores the database in memory only, with no persistence between runs.

@andrestaltz %Owyoh0varKNkBBUbL5FXSbWZxBclfAjr1+zfHU/QZVA=.sha256

And... for Android https://github.com/InstantWebP2P/node-android

@andrestaltz %/6kkPqt7pYWzZ3LTTqz/7ZlC+I5n3Bbr/66UOeA5DrI=.sha256

And gosh, this may help a ton, even though it's API doesn't match with fs, it's still an incredible asset to use: https://github.com/wkh237/react-native-fetch-blob

@andrestaltz %slPQV/ArADtyYuwSR7eNBrLIVELWEj5M8l/orZmvcJ0=.sha256

So I spent some hours trying that approach for scuttlebot (normal sbot as a background thread in ChakraCore in iOS, normal ssb-client in React Native) and basically I can't get it started because the background thread can't find native sodium and leveldb on iOS. I think there should be a way of getting https://github.com/mochtu/libsodium-ios into one of those npm sodium packages that bundle prebuilt bindings (should be an easy PR, right?) and same thing for leveldb, but this is far from my skills as I've never touched node.js native bindings.

@dominic some hints here?

I still want to push for this approach for a while since, once the native stuff are figured out, we can use our conventional approach of sbot server + ssb client with muxrpc to fetch data and display it, and it would be quite good as a basis for development since we wouldn't have to worry about performance due to a hacky stack of modules.

I could push the two efforts: the previous one for Android, and this new one for iOS. Whichever approach "wins" first in convenience and quality, we try to port to the other platform.

@Dominic %giGMNDagch3EXpJpKjR9m7YUOhcUp+SvN3ekC4osE9c=.sha256

@mafintosh just pointed me to a version of blake2b that he handwrote in web assembly (supported in node 8 and all browsers except IE) it's only 1/4 the speed of c sodium, and the output file is only 1.5k! possibly this is will just make crypto viable as is

User has not chosen to be hosted publicly
@andrestaltz %emMVMl9ULhl+GGaOEBMhomWs99yEKMbRYkVMHIsa0so=.sha256

Current status: trying to compile node-sodium for iOS. Managed to compile libsodium for iOS arm64 (haven't tested it though), but to make the file build/Release/sodium.node I need to fight a bit with GYP and C compilation flags.

@cel %BJxMxwpwJK27c2M2n+rG7yXWigvCendio5guDYCKvIE=.sha256

@tom: ssb-mobile roadmap

@andrestaltz %SdJZ4szR7rAGX/bKo+MPKYIKHOQyEJXyp82EqE553U8=.sha256

Super disappointed with a critical piece of software being closed source: https://github.com/janeasystems/node-mobile-react-demo/issues/1

Node.js is open source but they want to make Node.js-ChakraCore-iOS proprietary. It's not the first time I hit this wall when building something for Scuttlebutt, previously I bumped into http://underdark.io and OpenGarden MeshKit (how can you even name it OpenSomething if it's not open?).

@andrestaltz %VXCBYv2d7f2oUm29+liS0nAR2VUCcHRt4oX1hzW8caI=.sha256

Some good news: I'm investigating the possibility of doing that same iOS approach but for Android, and I found this: https://github.com/dna2github/NodeBase

So far, it's good! I got a full node.js process running in the background and it ran an http server which the React Native app was able to request.

@Dominic %ZGL+T4tIlC0/qib+2u/ZcoItrlHFfRQzQah3HYs6u3c=.sha256

That looks really good, and whats more, it's actively being developed! and there only a moderate amount of code to read!

in the readme it says:

Android NodeJS Platform to Build Sharable Application (Android as a Server)

Share application with your friends in the same Wi-Fi!

sounds like he's on a p2p vibe too!

@andrestaltz %xQ54A/GdHHA/EdWQ1qwd17lxcxh4MabaIgQ/UAEZG1Q=.sha256

Yep! And the node.js binary compiled for arm is v7, that's at least half awesome.

Update: I'm creating react-native-run-background-node-process (will find a better name) and hope it'll be a good foundation to build more stuff on top. E.g. next thing would be react-native-scuttlebot which just puts that one and scuttlebot together and it just works.

@Dominic %a2xE+2NTEvK/jgnr4gQE+g3QmlIF9tIIUMs8PjZMY64=.sha256

Hmm, I was under the impression that you couldn't start another process in a non-rooted phone, but it looks like he's just starting a process?

https://github.com/dna2github/NodeBase/blob/3976239d8e268622572dc18fba2f2f5d12e3d3fa/app/src/main/java/seven/drawalive/nodebase/NodeBaseServer.java#L117

Where does the arm node binary come from? is it the standard node arm binary? we gotta get sodium into that somehow.

@andrestaltz %9+z/ErJFntVdFeSYwZmP6+SDVh6k0XIoQ7/pU45rW40=.sha256

Yes, that line you highlighted is what does it, and currently in my react-native-run-background-node-process I have that same thing going on. The arm node binary comes from here: https://github.com/dna2github/dna2oslab/tree/master/android/build

And yeah, my current status is trying to compile node-sodium for linux arm. libsodium already has a make system for android, so that's done. Next is to get node-gyp to do the job correctly. Sailing uncharted waters here, no web searching seems to help.

@andrestaltz %6uM3az6vvV4kwLGZqJ4DQtyKdYM1mn2ZdoL0EyRdne4=.sha256

IT FREAKING WORKS

I don't know how else to say this: it works!!!

Basically I'm running a real node.js process in the background, which runs full scuttlebot (leveldown swapped for memdown, sodium is fully in JS, but fs and net and dgram and all the rest are the real shit) and the react native app uses ssb-client to communicate to that, and things... work.

Here's a video to prove it: https://instant.io/#3d7243c78252b94f6f803259123a64f9edefbf31
laptop can sync with phone and vice-versa. I'm also going to test phone-to-phone sync, but it should work too.

Next steps are getting the code published in different packages, and allowing each part to evolve separately. Obvious steps are getting an actual native libsodium and native leveldb, but those can evolve separately and then get plugged to the app later on. That's important so UI development can progress independently from the underlying infrastructure.

I'm super excited for this milestone

@dominic

@andrestaltz %U8/zH9uhpZyno2TYnWOSzhTVy6uvtXgEEBU1ztHlq0g=.sha256

phone-to-phone.jpg

Phone-to-phone sync over LAN, basically in real-time (judging by the naked eye, less than 100ms delay).

@andrestaltz %Ktm/4moAoyqeLAa2FWshuOp72NLpV6cF+RRL430UBSg=.sha256

PS: I like the background process idea because when it's managed as an Android sticky service, even if you close the app the process can still be running. We could use this to enable proactive data sync, so when you open the app you just browse the latest content as if it was always there. This could of course be user-controlled in some settings screen.

So exciting!

@Dominic %8Z5qMdeMNoyWUqnOngU0ut5AwzvmoGIHdrDqFbU+oY8=.sha256

WOW awesome! Can't wait to try this!

@andrestaltz %C0pY0rqjo2v2g60rW/6+QuWe4yulF/OMZRYe3teqQuU=.sha256

@dominic to get the repo reproducible for others, I'd need you to merge a PR for noderify first :)

@andrestaltz %sIg6kxzwJbPAMTi8vWy998f/TRX+FhmyAJQYSwECMCI=.sha256

The repo is http://localhost:7718/%25XnPjP9GWJ4z7W51Hh%2FUkc0aouHbY1AvxcQE5mkPLDwc%3D.sha256/tree/sbot

and branch sbot

User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
@andrestaltz %TbFDuvnRzd4nT31s2wQUX/sB4NjFIX3jZXPB7W2pdNg=.sha256

image-works.jpg

Images work too. I just put serve-blobs.js from patchwork and it hosted the images like any other web server does, and React Native <Image> knows how to consume that localhost URL. The nice plus is that you can also open the URL in the mobile browser! (Finally you can run a localhost node.js server on mobile easily)

@andrestaltz %35L3bxbDoidkmZ6Ni/5mjBkIyY/D6SGsCgCZ772t6G8=.sha256

Just published react-native-ssb-client https://www.npmjs.com/package/react-native-ssb-client

API looks like ssb-client, but is meant to be used in a React Native app. See my sandbox repo, branch sbot for an example usage. Basically just put "react-native-ssb-client": "1.0.4" in package.json dependencies, then in index.android.js, import ssbClient from 'react-native-ssb-client'; and use ssbClient like you usually would use the npm package ssb-client.

@andrestaltz %rLYCIHueMe+blQ0p/N2v44aq2dRhCz3zZao5IHfOvsg=.sha256

I just moved that to the ssbc org: https://github.com/ssbc/react-native-ssb-client

User has not chosen to be hosted publicly
@mix %hBWe1QZnlG6hBpWkCfv+2GcUZj7V7PnriE4oJYqaZT4=.sha256

wowowowow, very exciting, excellent work @andrestaltz

@andrestaltz %P35i7pv+gKbZ9NcXClgwwbRNx6w9ANooCh4huEjKejQ=.sha256

I kind of finished react-native-scuttlebot, once @dominic releases the new version of noderify, I can publish it.

Usage would be quite sweet: scuttlebot.start() then use react-native-ssb-client like we use ssb-client, both in index.android.js.

@andrestaltz %uAzDAODuDJcrW3ypCJ5QHlqi5QEw5Xg5dBH+IMCvhk4=.sha256

Here we go:

Use them in index.android.js like specified in their readmes

@andrestaltz %vj6h80x44tXwEzBXkPZKR1Dgtwb/XxOLTMvK2v7BQ9A=.sha256

See http://localhost:7718/%25XnPjP9GWJ4z7W51Hh%2FUkc0aouHbY1AvxcQE5mkPLDwc%3D.sha256/tree/sbot for an example

@cryptix %ja/zHPzVHzfE5uiMM8vLHvsuEivaLB9mTmT7dE2yfVY=.sha256

Lots of congrats, Andre!

User has not chosen to be hosted publicly
@andrestaltz %rAzVb7N18Zs8h6ZgLWMI0G73Si2gvWZ7fFbQomTu9MA=.sha256

@RubberDuck not yet, but once I do I'll certainly update this thread.

@andrestaltz %A5UC6H3+2KbKtA4kmhnlfBoujXx3y1krvEcMxELzBKY=.sha256

Today I got all the pieces in place, to start actually building the app: I have a RN app with the SSB modules I made, plus some framework stuff, some basic components, the app's GUI skeleton, icons, etc.

mmmmm-2017-07-18.jpg

The four tabs are:

  • News feed
  • Channels (?)
  • Metadata
  • Notifications ('mentions')

Metadata is a screen showing your connected peers, gossip data, a dashboard of ... metadata, etc. This will be a place where the user has access to raw data about everything they store on device. You add pubs here, you accept invites, etc.

I'm not sure if I should have Channels as a tab. Channels can also live in the menu accessible from the menu button on the left. Maybe I should have

  • Public
  • Private
  • Metadata
  • Notifications

Thoughts?

This was what I achieved today and tomorrow I'll start making the basic news feed screen with messages and publishing, hooked up to background-scuttlebot and using some patchcore primitives.

@mix %ShG7rdEk3wZi61GuHbu9DOg9KFrW9t9ZcNZP8eKfDPc=.sha256

@andrestaltz nice !

I prefer public / private / notifications / channels. I think meta is mainly useful for diagnostics, and for getting set up.

I guess initially users will be people who already have SSB identites on network. They can friend their phone identity and that will side-step need for onboarding (pub invite) functionality in this app. I know it's needed in the long run, but I'm a fervent believer in "just build enough to get people using the thing" and there's an active group of people who we can test with before going wider.

OOOOHHH boy, I'm pretty darn excited for this. If I even have read-only I'll be ready to pretty much drop FB

@andrestaltz %Ih45+pkORT9jK43HJoUqNUrqLHTrjj8HgSFs4ij2lH8=.sha256

Yeah maybe I should do public / private / ...

Don't get too excited just yet (I know screenshots have that effect :smile:), it really does nothing for now. Mega important features to build are: thorough and fully-supported messaging publishing and markdown rendering, but really: efficient storage. Just think how long it will take to sync with your desktop scuttlebot, if it even fits on your mobile device! :D

User has not chosen to be hosted publicly
@mikey %+aTU7XRdFwCUXx6AB4Sv95MdKEOufJVl0FqxVLsWbqo=.sha256

from the latest Patchwork (which i'm enjoying most):

Public / Private / More --- Profile / Mentions

patchwork-nav.png

where More has

  • Channels
  • Gatherings
  • Extended Network

patchwork-nav1.png

@andrestaltz %R7SLW9ZQ/Vdk9xGCezDY0nU+T6Ugtc3VZUGV+RnvN58=.sha256

Well, notifications can come from public or private, they are basically "mentions". Like I can mention you here like this @marco28. Also notifications will include a few other wildcard things like "Set up your profile" when you open the app initially.

User has not chosen to be hosted publicly
@andrestaltz %7LdWFdB0bFLBwoYltfnCRdiUaqHHw3yRBde0AnXqtrU=.sha256

I bring news:

mmmmm-2017-07-19.jpg

And repository: https://github.com/staltz/mmmmm-mobile

Milestone for today was using patchcore and sbot to fetch feed messages, and render them. The raw JSON box you see there will be displayed only for "messages of unrecognized type". Currently all message types are "unrecognized" but obviously it's just a matter of time until we build message components for each message type. The feed works. I tested it: with patchwork on my desktop, a new message appeared in real time on the phone.

Of course, consider this app to be a work-in-progress for a long time. I think I'll consider it "ready" when it's in the Google Play Store (unfortunately to reach a lot of people I need to go through that disgusting paywall).


@CustomDesigned Good point, SD card storage will be a must. Right now it's using the app's storage space, which is usually not in SD, but this type of configuration is possible, for sure.

User has not chosen to be hosted publicly
@mix %Z85/BHm8qQ7tjTN1/i+cl0snB8izAiodnL4RBfnC3NU=.sha256

hey @andrestaltz I'm not the sort to blow up with expectations. I mainly want to make sure you're getting enough positive vibes and encouraging feedback :)

I'm very realistic about the alpha nature of this, AND am really happy and willing to wade into testing alpha apps to support people doing rad work <3

User has not chosen to be hosted publicly
@andrestaltz %MvKBbKj8C5Z7VhxwHUADuTW0l1D6fhulPbl/Z5zsjgY=.sha256

The apk on github https://github.com/staltz/mmmmm-mobile/releases/tag/v0.0.1
cc @dominic

@andrestaltz %jSdT+213rtdNbHoRy8siUxd/ZrOIbkalqiQ6EwG/bSQ=.sha256

I got basic metadata like connectedPeers appearing in the app, using patchcore modules, but I had to pause to figure out how to display that data in the app in the correct way.

So I did some bit of designing in Sketch, and here it is:

metadata-design.png

The 1st screen (on the left) lets you choose to see either the 2nd or the 3rd screen.

Feedback?

@Gordon %+ljeSa8KBg/ZmKOwKkgnpWmV1GTloeMjKl1kES8xz9o=.sha256

Exciting :).

On the 'Peers on the internet' screen are the dots on the globe an accurate representation of where the node is based, or is it just a way to signal to the user that they are 'somewhere remote on the internet' and the location of the dots on the globe are not relevant?

@cryptix %Pa6aR/vxhxzvIXOYobR+3SuNpOMl5Rp/z5j4CMQBITY=.sha256

@happy0 my guess is, it uses geoip to map the location. So if the node uses some kind of vpn or proxy it could be anywhere but it's still a nice way to visualize it, I think.

@andrestaltz %5VIcI0Wr+Y9UX8BiC6WIf3r4pmzczn9Bb7QHRaPUolg=.sha256

It will (doesn't yet, it's just design) use geoip to pinpoint the location.

Here's an update: I coded the first version of the above. This screenshot is real data from my LAN:

device-2017-07-25-213832.png

@Dominic %CwhKspz6+SxYbcYhoJa94UTZNB9RGxTxuuj54LaeJbU=.sha256

looks great! what if you animated latency (peer.ping.rtt.mean) by showing dots moving between the peers?

@andrestaltz %WfVqsSVD7Rn9lcjJfYH1a4pzc8Z61WLnWMjdu0D/gbs=.sha256

@dominic I had planned to do that ;)

User has not chosen to be hosted publicly
@andrestaltz %dysxoBbN9mzmz5j0vQPyCCjosCAYQu7UKgTdxhaxOuA=.sha256

News

Markdown rendering

markdown.png


Simple compose message field, that actually works

write-public.png


Expand/collapse local peer metadata

expanded-metadata.png

@Anders %OOuIBj5M5y1AG/Pvpx0Xy/cjf97xNPKhLHscYrj8wm4=.sha256

Really could have used this on my vacation. Glad to see it coming along nicely!

@andrestaltz %FadQTAjy7NxJqPiWQfevu/NYzDjQUzxyquyRiHXk7pg=.sha256

version 0.0.2

https://github.com/staltz/mmmmm-mobile/releases/tag/v0.0.2-alpha

newest thing is like counter and like button

screenshot.png

@Anders %dUc83eo/bcv33Kp/6rdyq05YIgp8HkSB/c1W++L/hV0=.sha256

Just installed it and peers around you is working! How do I set up so I can see messages?

This is really exciting @andrestaltz!

@Anders %movgcKX8pYJDEDmz2wlB/saFM7NQjkl4x5/gdb6rfFI=.sha256

Seems to be as easy as following the local peer. Nice!

User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
@andrestaltz %Ng7PJE+ulXk43ssuyy+rsGLBIFgK8We5JflmOTNpVPk=.sha256

@keks That's good feedback, I'll try that. Also makes sense given that Twitter mobile app also has notifications tab as the 3rd tab, not the right-most tab.

@ub|k It should work if both phone and computer are on the same LAN. Make sure your phone's WiFi is on, and maybe reinstall the apk.

In general, this is very early alpha software. I'll label it beta once I start believing it's usable.

@dangerousbeans %g6ShyrSPObfp/UwOhreQZC2PNXJTIWm67puHfzVePTc=.sha256

Build for 7.1 on 32bit HTC m8 https://instant.io/#9648b7d2e061431cc97fd1437725b35d42755ae3

worked great, just got stuck when I was running weird react-native commands instead of just doing npm run

I think it's actually syncing faster than my macbook pro did first time round

User has not chosen to be hosted publicly
@cel %39p6Lj6hmT8nDkzg+FHmJYjODa6URplGLw8re3YUqr8=.sha256

@ub|k scuttlebot broadcasts its address on UDP and detects addresses it finds broadcasted, adding them to the gossip table. the actual SHS connections are unicast.

User has not chosen to be hosted publicly
@cel %6qMOJZwJDrcWsm9cgOE7Kxmp2gJek8XRSG0riquu6wo=.sha256

@ub|k
with zeroconf, peers broadcast queries and respond to queries that they see broadcasted. this is a more standard way of doing local peer discovery.

sbot local peer discovery is basically just hello packets repeatedly broadcasted (every 10 seconds) - there is no query/response.

UPnP is used to tell the router on your network to forward a port to you from the WAN. eventually we might want to use this to let home nodes automatically become pubs.

User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
@andrestaltz %52QLUckf/f67REMZWVhZeu+a7vreJqXDYiHEzO0wFmk=.sha256

Great to hear @n8fr8! That's awesome, I'm aware of Orbot.

I'm not sure if you know about this already, but I think SSB supports Tor too, inside multiserver: https://github.com/ssbc/multiserver#onion

About SSB on mobile, we need help. Most important problems to solve are:

Both leveldb and sodium/chloride are huge important pieces of the infrastructure that currently aren't working at all. (we use RAM for "storage" and slow crypto-in-js)

:)

User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
@xj9 %Fhx37wkqi0VLUsw8zjL8uuoTCx6aQPqq6GSmdVqjwuM=.sha256

@crasch the model i'm using for my p2p app toolkit is something i'm calling "anonymous pseudonymity". basically, your identity is an asymmetric keypair that can't be linked to a specific physical device, network address, or location through normal coms so, if you want strong anonymity you can discard your key after every session and/or keep certain keys around to provide a reference persistent or semi-persistent identity.

https://journal.xj9.io/posts/2017/what-am-i-doing-with-my-life/#router

User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
@SoapDog %TIx8XA1lBacRrt2ILs/qKVNdB5v7qEeXHuSvaj7J8U0=.sha256

Hey @andrestaltz been trying to learn more about #ssbc, projects and the grants system and something got into my head which I don't know how it could work. My main doubt is how can you have the same identity in more than one machine since we can't have forks/merges of the log. The mobile version would use a different identity than the desktop/home version?

User has not chosen to be hosted publicly
@andrestaltz %f+Cr701vsAe2ni6t1neLpDS2yArpPg1rvQ1TpYrrK+g=.sha256

@SoapDog, check this: %JjSO5d5...

@SoapDog %21o6tzNhsBSZNl4zQUaaJ0D6Y6T/4Y4BATN+obuw8A4=.sha256

thanks a lot, that is much simpler (as in more straightforward) than I imagined. I thought about breaking changes to the protocol. Glad to learn that it can all be solved on an upper level.

User has not chosen to be hosted publicly
User has not chosen to be hosted publicly
Join Scuttlebutt now