You are reading content from Scuttlebutt
@Dominic %iqXoP//+WnBt3UhTKK2uKjlV4EojMn8IjY5/rSLKdPY=.sha256

ssb-tunnel

prompted by recent work on webrtc and utp support from @piet and @andrestaltz I spent some time refactoring how secret-stack handles protocols, some details still to be figured out, but sbot plugins will be able to provide new protocols.

With that proposal in place, when I had a bit of time spare I took a stab at implementing another one: what I called "relay p2p" in the TechTree - ssb-tunnel it complements the other p2p methods, by using a pub as a relay - A connects to the pub, then B connects to the pub, then B requests a tunnel from the back to A, then treats that as an ordinary connection, authenticated and encrypted with secret-handshake. This means the tunnel connection is encrypted twice and the pub can't see what you are saying.

Again, a few small details to work out still but I have a working proof of concept!

The really exciting thing, though, is that this (and the work andre and piet are doing) is the critical step to bootstrapping other protocols.

For example: if you can create a stream directly to another ssb identity (which may be tunneled through pubs or whatever as necessary) you could have a chat application that had it's own protocol, without using ssb messages (...or something you could model as a chat application, say a game or collaborative editing)

It feels we've set up a lot of dominos, and once we have private groups, most of the protocol features I've been thinking about will be roughly in-place.

@andrestaltz %6wtwU+N56v4sheCpR7JAtWuq8xSHKngHCFadfLF4P+Y=.sha256

Smart that ssb-tunnel uses multiserver style addresses! Also makes sense from a social perspective: "connect directly to me @7MG1hy when I'm online in the pub scuttlebuttia.nz"

User has not chosen to be hosted publicly
@Dominic %lDeocx/1I7DUTGLeybuUfbKlSJVLJ/pJ7shRLRQJY58=.sha256

@andrestaltz of course! then tunnels are just another type of connection, and so anything that works over any other protocol can work through a tunnel.

@Tim-Makarios the answer is both. ICE style still needs to pass some coordination between peers, so could could use the STUN/relay style to bootstrap the ICE/direct method.

@Dominic %DwUOzE3umr3Hve5kVfl7XYjcpxQQylvufKfUSTaKr+Y=.sha256

thinking about this again, after this PR

it would make more sense to specify tunnel addresses as tunnel:portal:target than tunnel:<portal_address>:target because the target shouldn't have an opinion on how you connect to the portal. Maybe you can only connect to them via websockets?

On 3rd thoughts, maybe the address should be tunnel:hmac(portal, target) since you have to know a connection address for the portal anyway, this now means the identity of the portal is not revealed (unless it's already known). That would enable fully private pubs - provided we also had implemented privater-gossip-requests

@Dominic %MUIB6g7rRs0+73PxMb3poWhQ4X2MT2M41Wb64BZA4wU=.sha256

I spent some time on this again today - got it working again after @arj's multiserver plugin stuff.
I think this is ready to be deployed for proper testing... I anticipate problems with connections to pubs dropping though, but that is just something we havn't needed to look at in a while.

@Dominic %vgsxzYZUfcbBn6NMHdaod/JbiptNNFTXlfcRQRznfUY=.sha256

Okay! I got a connection tunneled through my pub! if someone else would like to help out here, see if you can make get ssb-tunnel working (check out, run tests, then follow instructions in example.js) then we'll be ready to test a connection between us!

@Dominic %Au+mVknHifAQluCtSwX98f8UwCWceiVmvmchkVBZMyQ=.sha256

btw, you'll need latest scuttlebot@12.2.2 and ssb-tunnel@1.1.0

@mix %T+0SzHsFpiKHgmxuDC+UfU5HOFv/w51aspmGkqV0d9Y=.sha256

I pushed a .gitignore

couldn't get the test passing, npm test produced:

> set -e; for t in test/*.js; do node $t; done

TAP version 13
# carol tunnels through bob to get to alice
failed to connect to portal: undefined
connected @7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=.ed25519
/home/mix/projects/SSBC/ssb-tunnel/node_modules/pull-stream/sinks/drain.js:22
                throw end
                ^

Error: iterator has ended
    at Iterator._next (/home/mix/projects/SSBC/ssb-tunnel/node_modules/leveldown/iterator.js:51:18)
    at Iterator.AbstractIterator.next (/home/mix/projects/SSBC/ssb-tunnel/node_modules/abstract-leveldown/abstract-iterator.js:27:8)
    at Iterator._next (/home/mix/projects/SSBC/ssb-tunnel/node_modules/encoding-down/index.js:109:11)
    at Iterator.AbstractIterator.next (/home/mix/projects/SSBC/ssb-tunnel/node_modules/abstract-leveldown/abstract-iterator.js:27:8)
    at ReadStream._read (/home/mix/projects/SSBC/ssb-tunnel/node_modules/level-iterator-stream/index.js:25:18)
    at ReadStream.Readable.read (/home/mix/projects/SSBC/ssb-tunnel/node_modules/readable-stream/lib/_stream_readable.js:443:10)
    at resume_ (/home/mix/projects/SSBC/ssb-tunnel/node_modules/readable-stream/lib/_stream_readable.js:790:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
@Dominic %5qfZW2s5RY35vlG0SW+PK3hw5+tjaszd11cwAI/Hit0=.sha256

@mix hmm, I am unable to reproduce with a fresh install. Can you try again? (pull again to get fully up to date deps)

If it's still broken, please post some version of pull-level, level, and leveldown, I get:

> $ npm ls pull-level level level-down
ssb-tunnel@1.1.2 /home/dominic/c/ssb-tunnel
`-- scuttlebot@12.2.2
  +-- pull-level@2.0.4 
  +-- secure-scuttlebutt@18.3.3
  | +-- flumeview-level@3.0.6
  | | +-- level@3.0.2  deduped
  | | `-- pull-level@2.0.4  deduped
  | +-- level@3.0.2 
  | +-- level-sublevel@6.6.5
  | | `-- pull-level@2.0.4  deduped
  | `-- pull-level@2.0.4  deduped
  `-- ssb-blobs@1.1.5
    +-- level@3.0.2  deduped
    `-- pull-level@2.0.4  deduped
@mix %km9M6Gbgv4PpD1PIiTNaOpbudQifZ2yKLtC43V/Sqns=.sha256

same error.

I'm on commit : a363d4a29968f6f297f838ea8224d1361a216b89 (tag 1.1.2), my patchbay is running while I run tests, incase that matters.
node -v 8.11.3

here's the npm ls output :

ssb-tunnel@1.1.2 /home/mix/projects/SSBC/ssb-tunnel
└─┬ scuttlebot@12.2.2
  ├── pull-level@2.0.4 
  ├─┬ secure-scuttlebutt@18.3.3
  │ ├─┬ flumeview-level@3.0.6
  │ │ ├── level@3.0.2  deduped
  │ │ └── pull-level@2.0.4  deduped
  │ ├── level@3.0.2 
  │ ├─┬ level-sublevel@6.6.5
  │ │ └── pull-level@2.0.4  deduped
  │ └── pull-level@2.0.4  deduped
  └─┬ ssb-blobs@1.1.5
    ├── level@3.0.2  deduped
    └── pull-level@2.0.4  deduped
@mix %a6uGEvRzQTIN3iuB5xUh/Qq7UAbR5yJA21TtiAlCseU=.sha256

npm 6.3.0

Are you gitignoring your package-lock, or not commiting it?
I think having the present might make it easier to debug dep differences no?

:eyes: creepy whisper : maybe just sneak it into a branch

@Dominic %7XuQlyAoc/lPK30steSB7zUc7MTvmOv1Y3+5gwsAjuk=.sha256

@mix we don't know that it's a dependency problem... my hunch: it's a race condition around closing the database. confirmed: I managed to reproduce it in the flumeview-level tests...

@Dominic %dlUaBU5yqn5CQbmzeFQ7HO/2pja5sbXnehdfDhs6b9M=.sha256

@mix okay i realized there was a simple solution - should be fixed in scuttlebot@12.2.3, check out latest ssb-tunnel and reinstall.

@mix %qtDksLy3hw2UqeLef91ngMuLKDuW6PzUjKaAcUYeZI8=.sha256

@dominic still failing tests after burning them all and starting fresh

git head : 038f503998d8799490ef122aefd3c9fe5691f04d (tag 1.1.3)

TAP version 13
# connect two peers - just to test connections config
failed to connect to portal: undefined
failed to connect to portal: undefined

1..0
# tests 0
# pass  0

# ok

TAP version 13
# carol tunnels through bob to get to alice
parse: net:192.168.1.162:3005~shs:7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=
ALICE??
parse: net:192.168.1.162:3005~shs:7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=
>>>
Error: stream is closed
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/index.js:45:25
    at remoteCall (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/api.js:40:19)
    at Object.obj.(anonymous function) [as announce] (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/api.js:25:18)
    at /home/mix/projects/SSBC/ssb-tunnel/index.js:93:26
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/scuttlebot/plugins/gossip/index.js:212:17
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/secret-stack/index.js:204:36
    at next (/home/mix/projects/SSBC/ssb-tunnel/node_modules/multiserver/compose.js:28:14)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/multiserver/compose.js:34:9
    at _cb (/home/mix/projects/SSBC/ssb-tunnel/node_modules/multiserver/plugins/shs.js:31:11)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/secret-handshake/protocol.js:120:7
ok 1 should not be equal
CAROL CONNECT tunnel:@7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=.ed25519:@1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=.ed25519~shs:1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=
TUNNEL.client { name: 'tunnel',
  portal: '@7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=.ed25519',
  target: '@1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=.ed25519',
  port: 0 }
TUNNEL.client @7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=.ed25519
attempt tunnel connection to: { name: 'tunnel',
  portal: '@7MG1hyfz8SsxlIgansud4LKM57IHIw2Okw/hvOdeJWw=.ed25519',
  target: '@1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=.ed25519',
  port: 0 }

/home/mix/projects/SSBC/ssb-tunnel/test/tunnel.js:77
      if(err) throw err
              ^
Error: could not connect to:@1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=.ed25519
    at DuplexError (/home/mix/projects/SSBC/ssb-tunnel/index.js:6:13)
    at Object.connect (/home/mix/projects/SSBC/ssb-tunnel/index.js:150:16)
    at Object.hooked (/home/mix/projects/SSBC/ssb-tunnel/node_modules/hoox/index.js:10:15)
    at Object.localCall (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/local-api.js:31:29)
    at Object.<anonymous> (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/local-api.js:37:22)
    at PacketStreamSubstream.stream.read (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/stream.js:67:23)
    at PacketStream._onstream (/home/mix/projects/SSBC/ssb-tunnel/node_modules/packet-stream/index.js:228:11)
    at PacketStream.write (/home/mix/projects/SSBC/ssb-tunnel/node_modules/packet-stream/index.js:135:41)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:56:15
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/pull-stream/sinks/drain.js:24:37
@Dominic %9CRyVjXhnMF/RiRqkfADLpnBeP8nv0GjeP06V1bNTRQ=.sha256

@mix I removed a suspected race condition from the tests - can you pull 1.2.0 and try again?

@mix %Y1D43I2u028A1gtnKvX96q5ir5SlUagra7rlz+YJe88=.sha256

4/4 passed

and there was this exception in the middle ?

Exception thrown by PacketStream substream end handler Error: callback not provided
    at noop (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/api.js:14:18)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:18:14
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:10:5
    at source (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:44:20)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:24:15
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:10:5
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:59:17
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/pull-stream/sinks/drain.js:20:24
    at PacketStreamSubstream.weird.read (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:33:7)
  Error: unexpected end of parent stream
    at PacketStream.destroy (/home/mix/projects/SSBC/ssb-tunnel/node_modules/packet-stream/index.js:71:7)
    at Object.ws.close (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/stream.js:134:23)
    at EventEmitter.emitter.close (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/index.js:78:8)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/scuttlebot/plugins/gossip/index.js:134:16
    at Array.forEach (<anonymous>)
    at EventEmitter.<anonymous> (/home/mix/projects/SSBC/ssb-tunnel/node_modules/scuttlebot/plugins/gossip/index.js:133:26)
    at EventEmitter.<anonymous> (/home/mix/projects/SSBC/ssb-tunnel/node_modules/hoox/index.js:3:17)
    at EventEmitter.hooked [as close] (/home/mix/projects/SSBC/ssb-tunnel/node_modules/hoox/index.js:10:15)
    at /home/mix/projects/SSBC/ssb-tunnel/test/tunnel.js:84:13
    at Object._requests.(anonymous function) (/home/mix/projects/SSBC/ssb-tunnel/node_modules/packet-stream/index.js:38:5)
Error: callback not provided
    at noop (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/api.js:14:18)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:18:14
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:10:5
    at source (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:44:20)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:24:15
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:10:5
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:59:17
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/pull-stream/sinks/drain.js:20:24
    at PacketStreamSubstream.weird.read (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/pull-weird.js:33:7)
  Error: unexpected end of parent stream
    at PacketStream.destroy (/home/mix/projects/SSBC/ssb-tunnel/node_modules/packet-stream/index.js:71:7)
    at Object.ws.close (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/stream.js:134:23)
    at EventEmitter.emitter.close (/home/mix/projects/SSBC/ssb-tunnel/node_modules/muxrpc/index.js:78:8)
    at /home/mix/projects/SSBC/ssb-tunnel/node_modules/scuttlebot/plugins/gossip/index.js:134:16
    at Array.forEach (<anonymous>)
    at EventEmitter.<anonymous> (/home/mix/projects/SSBC/ssb-tunnel/node_modules/scuttlebot/plugins/gossip/index.js:133:26)
    at EventEmitter.<anonymous> (/home/mix/projects/SSBC/ssb-tunnel/node_modules/hoox/index.js:3:17)
    at EventEmitter.hooked [as close] (/home/mix/projects/SSBC/ssb-tunnel/node_modules/hoox/index.js:10:15)
    at /home/mix/projects/SSBC/ssb-tunnel/test/tunnel.js:84:13
    at Object._requests.(anonymous function) (/home/mix/projects/SSBC/ssb-tunnel/node_modules/packet-stream/index.js:38:5)
tunnel:connect - failed to connect to target:@1b9KP8znF7A4i8wnSevBSK2ZabI/Re4bYF/Vh3hXasQ=.ed25519 unexpected end of parent stream
@mix %Pf7RGF28xnoGr3E32TnN7IT+7GNY9OAb7leGYdGsxbI=.sha256

cc @dominic

@Dominic %pFbyqbbbBOfljdIgrkfcI8TvMws9iBv03hDVGHe92Y8=.sha256

@mix great! you can ignore that other stuff.

@mix %ao9StAjhQ2Qj9QKIxeto3UuV9wHu2RhNluZezu6Smeo=.sha256

you're welcome @dominic
let me know if there are any other things you'd like simple help like this with

@Dominic %IhQ26L7Q2Z21s+M9Ctj1j20BxbwT2Ay5iGMbP1hxYkw=.sha256

Update: writing more tests for ssb-tunnel using ssb-public (todo: rename) where tunnel end-points are added to the peer table. Important: they can also be announced privately! I don't want the whole world connecting directly to my laptop, but my friends are okay.

And I discovered lots of problems in scuttlebot/plugins/gossip and ssb-ref - they both currently assume that addresses are convertible to the legacy host:port:key format (not true with tunnel addresses!). That is gonna take a bit of tidying up...

@mix thanks, will do.

@Dominic %mzPbHnh8N9TsnFNNl6pQFw4xVCCalVrrsm48H1oMHT8=.sha256

Hey everyone, I got the ssb-tunnel tests working again with the latest versions of everything (lots of changes happened in sbot since I first wrote ssb-tunnel) I still need to get the examples working, but that should be easy now.

@mix %PNlIHRlyt2fU2eOoAtjw3xn0B5M5+CYPrHRp+qLot4w=.sha256
Voted Hey everyone, I got the ssb-tunnel tests working again with the latest vers
@andrestaltz %+k0ObHEWuIisYWQa8UPuBdgVesZlUimnnxjAE7q4+H4=.sha256

@dominic sweeeeet

@Dominic %HkPDQKQpW4e550n7h4h+FHcZB+KtI1cg9XWgcghOwWQ=.sha256
Voted [@dominic](@EMovhfIrFk4NihAKnRNhrfRaqIhBv1Wj8pTxJNgvCCY=.ed25519) sweeeeet
@Christian Bundy %bbauJpE+d+nGb2haLOfkbdAcqsWSlJL8CfKeedgwT4w=.sha256

@dominic

Do you think it would be useful to create an overlay network like cjdns to connect peers directly via ssb-tunnel? I was just reading this about how cjdns does address allocation:

Address Allocation

The ipv6 assigned to the TUN interface is composed of the first 16 bytes of the SHA512 hash of the SHA512 hash of your public key. Keypairs are generated via a brute force method until a corresponding address is found with a starting byte FC. The FC00::/8 address space has been allocated as a Unique Local Address space, and so these addresses should not conflict with ICANN assigned IPv6 addresses or any other conventional internet operation.

Maybe it would be fun to join the fc00 cyber-squat?

Join Scuttlebutt now