ButtCloud dev diary

#buttcloud

grant proposal: %HqwAslt...

GitHub mirror: https://github.com/buttcloud/meta/blob/master/diary.md

i started early!

2018-03-29

• setup ButtCloud.org and various linked services (GitHub, Docker Cloud, Twitter, Gmail)
• started buttcloud/meta repo to store any meta information (including these dev diary entries)
• started `buttcloud/butt-landing to be a simple public landing page for pub servers
  • working towards a multi-service pub using Docker Compose, i realized ssb-viewer and git-ssb-web were non-trivial to install in typical environments
  • the plan is to use this as a simple secondary service to get a multi-service pub working, then later can focus on the other secondary services

2018-03-30

• made Docker image for buttcloud/butt-landing
• extracted minimal peer server and client code from scuttlebot into buttcloud/butt-peer
  • added custom logging plugin which uses pino, to try and have a consistent logging system across services
  • made two Docker images, one for butt-peer-server and one for butt-peer-client
    • i combined them so i could use the client code in the server healthcheck

2018-04-03

• started buttcloud/butt as docker-compose.yml of peer server, landing server, and nginx proxy
• update butt-peer-server and butt-peer-client to not use node user, easier to start with default root user
  • why? i ran into an error with volume data permissions, would rather punt to later
  • UPDATE: changed this back, got it working with node user, needed to create volume in Dockerfile then mount external volume at same path
• update butt-landing to auto re-connect to sbot, so it doesn't error when sbot is not yet up
  • why? this is the recommended way to have docker services depend on each other, docker-compose.yml v3 not longer supports depends_on: condition: healthy
• setup Docker Hub automated builds to build tagged images based on git version tags
  • match tag name: /^v[0-9.]+$/, docker tag is same

2018-04-04

• got minimal buttcloud/butt working!
• iron out some kinks...

2018-04-08

• switch to focus on Docker Swarm
• change to use Traefik instead of Nginx Proxy

2018-04-09

• keep trying to get the Swarm setup to work, grr...!

2018-04-13

• take a break from Swarm for meow
• huge leap towards scaffolding buttcloud/buttcloud-provider
  • have a working web server, browser app, and worker, but not yet complete to start feature development

2018-04-14

• small steps of progress toward provider app scaffolding
  • implemented the all important Facebook login button!

@Karsten shrug, i've had bad experiences with Kubernetes and i'm afraid of Mesos. i'm familiar with Docker, i like the slow development towards a set of orthogonal systems, i'm not a fan of the blind hype towards the new generation of deployment systems. what's your experience like with these systems?

is that FB picture for REALZ?

@Connor this diary is for serious grant updates ONLY, no subtle butts here!

context: to be clear, i want to have fun with this grant, so this diary will include subtle jokes like a Facebook login. please feel free to get amongst!

2018-04-15 to 2018-04-17

yay more progress on the provider app!

in the upcoming version of Patchwork, you should see a video embed above.

in the meantime, below is a normal blob link: 2018-04-17-buttcloud-landing.webm

• integrate redux-bundler
• found seamless background image to tile on landing page
• start onboarding workflow
• implement start step of onboarding
  • validate forms on client
  • validate service calls on server, show errors in form
  • show success or failure messages as snackbar
  • after form submission (which creates the user)
    • generate json web token that identifies user
    • send welcome email to next page of onboarding with token
      • emails are sent by queuing a delayed job to a worker (node-resque)
      • setup decent email templates with mjml
    • store user in local storage in case they refresh page before progressing
    • show help text on page
  • allow user to resend onboarding email

oh also, emojis!

2018-04-18

back to the infra side, during breakfast this morning i finally figured out why buttcloud/butt was failing!

i had a hunch that it had to do with the address that sbot was binding to. i configured host as example_butt-peer-server, since that's how the Docker service was to be identified within the Docker network. but still, the health checker inside the service couldn't find it. i changed this to 0.0.0.0 and it works!

did the same for the landing service. now the stack comes online, you can curl -H "Host: example.butt.nz" localhost and get the output from the landing page associated with example.butt.nz (proxied by traefik).

next i added a custom plugin to butt-peer-server that allows you to configure externalHost, in case it differs from host. this means we can bind to host (like 0.0.0.0) but advertise our public multiserver address as example.butt.nz (like for invites).

then, on a whim from @mischa, i went back to buttcloud-provider to swap redux-form for final-form, easy as.

made up some issues, want to step back to think about the next steps from here.

oh also, have yet to mention...

ButtCloud has an apprentice!

i've accepted @austin to be my "apprentice" for the next 3 months of the ButtCloud project.

This might be a bit of an odd proposal, and assuming you get the funding you need and decide to continue on with it, I was curious to know if you'd be willing to take on an "apprentice" during the project to help with code, documentation, etc. In exchange I could contribute an additional $500(USD)/month over a 3 month period for a total of 1500 USD to help fund the work (I could pay in crypto or cash, whatever is easiest). I would want NO stake in whatever business you'd like to establish moving forward. What I'm really looking for is a learning opportunity in a real world project with an adviser who can provide feedback on the work that I'm doing. I'm a self-taught NodeJS developer, but having never had a "real" development job, no one has ever reviewed my work or provided any feedback, so that's something I'd really appreciate.

i'm personally honored that anyone would give me their attention as if i'm a worthy "master", i'm already impressed by what's happened so far, looking forward to see how this progresses!

made the ButtCloud logo!

catching up on my diary!

some days are missing because partying at chur burn, doing other Root Systems work, doing a contract with @Mischa one-day-per-week, and bringing home a cold.

2018-04-19

• setup contributor license agreement: https://github.com/buttcloud/meta/issues/6
• setup kanbans
  • dev
  • ops
  • biz

2018-04-26

• start to separate pub and hub stacks in swarm setup: https://github.com/buttcloud/butt/commit/426deb39b9880100fe82ba5960da3d43fe74c452
• worked on deploy for web app demo: https://github.com/buttcloud/buttcloud-provider/pull/9
  • browser code is up at: https://demo.buttcloud.org (using netlify for free)
  • api server is up at https://buttcloud-demo.herokuapp.com/ (using heroku for free)

2018-04-27

• discovered and documented bug with tinyify: https://github.com/browserify/tinyify/issues/10
• add standard style setup to web app: https://github.com/buttcloud/buttcloud-provider/pull/10

2018-04-30

• demo is now live! demo.buttcloud.org
• renamed sub-projects to either buttpub* or butthub*, to standardize language: https://github.com/buttcloud/meta/issues/7
• setup continuous integration for butthub-provider: https://github.com/buttcloud/butthub-provider/pull/11
• setup ButtCloud account with OVH
  • apply for their startup support program (for maybe $1k cloud credit): https://www.ovh.com/world/dlp/
• play with docker-machine to create a local swarm across many machines
  • get the swarm scripts from buttpub working, now across multiple virtual machines

2018-05-01

• started docker-up: opinionated glue to manage our Docker swarm

2018-05-02

continued with docker-up

• ended up making a fun little continuable (cb => {}) based async flow control library in ./util/async.js, maybe will publish as callstep or flowstep or something
• realized that the Docker API doesn't handle the docker stack * functionality, that's implemented in the Docker CLI
  • i learned that a "stack" is really a set of networks, volumes, and services each with a label "com.docker.stack.namespace" to reference the stack name
  • have to decide whether
    • a) to continue using the Docker API and implement that functionality ourselves
    • b) to move to using the Docker CLI
  • for now, will go with option a) !
    • reading the Docker CLI code, it's not scary or complex
    • this way we have more low-level control of the Docker Swarm
    • this way we can focus on exactly what we need for ButtCloud

Just tested the demo -- looking great!

---

It has a nice look to it, simple and minimal

thanks for trying out the demo @ike @moid, good to hear it's okay!

Not sure where it's at, but I tried the demo. It captured my data, sent me an email, I clicked in the email and was taken to the setup step and a blank page.

@moid sweet, that's the expected behavior right meow!

2018-05-03

more docker-up, getting close to v1!

• add executable cli
• clean up the api
• fractal stacks!
  • top-level config is a stack, with stacks all the way down
  • each stack has services, networks, volumes, AND NESTED STACKS
  • each stack may have a name to namespace associated services
• pretty configurable logging

next up (notes to self):

• better cli (take in resource type)
• use explicit docker version in api requests
• add "com.docker.stack.namespace" label to be legit docker stack

gotta work with @Mischa on another contract meow, then Art~Hack!

2018-05-04

• reviewed @austin's sweet pull request to butthub-provider: https://github.com/buttcloud/butthub-provider/pull/13
• completed the "next up" features for docker-up in the previous entry

2018-05-07

• published docker-up/util/async.js as callstep, wrote up a splash of documentation

2018-05-07

• docker-up: add basic integration and unit tests using ava, clean up log and config wrappers using composable callsteps: https://github.com/buttcloud/docker-up/commit/29655b561b781331fc4b3a2455ea15e7360bb111
  • cc @ike
• butthub-provider: look into adding integration tests using codecept
• (not ButtCloud) improving ssb-pub for TickTack: https://github.com/ahdinosaur/ssb-pub/pull/10

2018-05-09

• butthub-provider: add end-to-end acceptance testing with codecept: https://github.com/buttcloud/butthub-provider/pull/14

2018-05-10

• butthub-provider: battled some end-to-end testing dragons : https://github.com/buttcloud/butthub-provider/pull/14
  • integrated the entire web app stack (api server, asset server, worker, and mailer) in the codecept process
  • at the end of the tests, the process was hanging, who was still running?
  • with some help from why-is-node-running and heaps of reading dependency internals, started the journey to find every remaining handle, gotta catch 'em all!

2018-05-11

• butthub-provider: won the end-to-end test war: https://github.com/buttcloud/butthub-provider/pull/14
  • cleaned up every last handle, so the test process cleanly exists meow
  • which means continuous integration now includes end-to-end server + browser tests!
    • travis gives us a running postgres and redis database, how nice
• butthub-provider: finish a boring dependency upgrade: https://github.com/buttcloud/butthub-provider/pull/12
• docker-up: https://github.com/buttcloud/docker-up/pull/4
  • the plan was to add update for services, but along the way i realized many things were broken:
    • each resource uses a different identifier!
    • down should check if resource exists before remove
    • up returns output of inspect
  • generic resource creator now takes:
    • name
    • hasUpdate: true for service, false otherwise
    • idField: network uses Id, volume only hasName, service usesID`
• docker-up: add continuous integration tests: https://github.com/buttcloud/docker-up/pull/5
  • travis even gives us running docker to play with!

2018-05-15 - 2018-06-09

despite my lack of updates, i'm still spending most of my time working on this, so much to do

spent almost a month (!) re-writing and improving the docker swarm manger, now called gyne.

• am starting to finally understand some advanced fp (functional programming) concepts like monads, in particular the value of a maybe type, a result type, and a future type.
• docker-up: investigate using sanctuary
• docker-up: port code to use ramda, folktale, folktale-validations, and fluture
• docker-up: re-architect how everything works
  • given next config, translate into docker api config
  • fetch all current docker api config
  • diff docker api current and next configs
  • show diff to sysadmin
  • if acceptable, translate the diff into docker api commands and run those
• rename docker-up to gyne, the insect queen
• rename ButtCloud to PeachCloud, purchased peach.cloud and peachcloud.org
• get example swarm working across 3 virtual machines using docker-machine and gyne

uh, sorry for the radio silence, i ran out of steam to work on this.

my latest plan was to ditch the product and push out a minimal viable tool: a peachpub command-line tool that would allow you to easily setup a pub swarm on your own cloud machine, using all the work i did on Docker et al.

but now i've had a realization, i've been thinking about PeachCloud all backwards. it shouldn't be a cloud-based pub-as-a-service platform, i don't want to be a centralizing force on the ecosystem, i don't want to deal with the associated regulations for hosting your butts, i don't want to use someone else's computer in a far away data center, i had my mind all turned around.

so, the new approach: PeachCloud is a hardware product.

i don't plan to have a fancy product release anytime soon, but here's a rough outline of next steps, should i find the energy:

• develop a better implementation of ssb-server that can reliably run with less resources, i.e. Sunrise Choir #sunrise-choir
• build a Raspbian image with Scuttlebutt bits installed
  • let you configure the image with your local wireless details so the Peach can auto-connect on boot
  • use Systemd to startup and manage services
  • setup ssb-server with some fun plugins: git-ssb etc
  • create a plugin to host a local web page for the admin to interact with the device: generate invites, keep track of invites, monitor memory / disk usage, etc
  • use nginx as an http proxy to host a web page on port 80
  • use mdns to advertise device at peach.local
  • use something to signal your dynamic IP to a dynamic DNS provider (maybe a service we provide)
  • (optional) also host a dat server, with associated web page interface
• play with Raspberry Pi hardware
  • install/update image on SD card
  • use standalone USB storage for uxer data (so your uxer data doesn't get mixed with the Peach disk image, updates should be much easier this way, don't really want to do fancy auto-updates over the net)
  • add Real-Time Clock to keep track of time (so you can be off-grid and still know the time)
  • add visual interface (LEDs, OLED screen, something to show the uxer what's happening inside the Peach)
  • add active interface (buttons, something to let the uxer interact with the Peach)
  • (optional) add enough solar and battery power to survive in the wild
  • (optional) create a waterproof case to fit all the yum inside and keep all the yuck outside, with a sweet peachy logo
• setup supplementary product services
  • dynamic DNS provider to route peach.cloud/myPeachKey to your dynamic IP
  • key backup (although i think Dark Crystal is doing this better than any centralized service could)
  • knowledge base: hardware design, bill of materials, how to build, how to setup, how to update, faq, etc
  • sell pre-assembled Peach devices

if anyone is keen to jump on this train to help, keen to support.

Yes!!

Hardware businesses are mostly decentralization friendly: laptops, phones, external hard drives, etc. I wrote about this last year:

Businesses that sell hardware will support the new decentralized internet, increasing people’s self-sufficiency and (literally) empowering them instead of exploiting them.

https://staltz.com/layers-of-the-internet-economy.html

I'd like to see "-as-a-Box" be the new "-as-a-Service". In particular, have you thought about CJDNS-as-a-Box?

Could get inspired by (if it isn't already)/integrate with https://freedombox.org/