You are reading content from Scuttlebutt
Feed of @jer

Libertarian. Digital designer and software developer. I post about politics, digital design, software, crypto currencies, hell, just about anything I feel like, even controversial subjects when I'm looking to learn something.

@jer %QjGkbUT0TN/MyKDasuC0nlrve2AoOQFo8LTni/AOBhI=.sha256
Re: %PeXe+6l4s

Saša is always talking about interesting things

@jer %3zyplp86n1Lcyr6+czOMovflQS4QdMp+/qpAp5yWmLU=.sha256
Re: %OXUt8kFZl

It's not just the US's laws. The US has been using it's 1 year posting as the head of FAFT to undermine digital currencies in general. For instance, FAFT now requires members to implement laws that require digital currencies REQUIRE not just KYC, but also for each party in the exchange to be conveyed the personal information of the other transacting party, for reporting purposes in their own country to combat money laundering.

There are 13 agencies in the US who are tasked with fighting money laundering; when one doesn't work completely, they just add another. Regulation is definitely killing things as governments try and maintain control of currency.

@jer %Yfd6eyr586brb5+WxsXAGRlvsdnoqd7dONCNSmdl+Eo=.sha256

What lessons can we glean from the original Ghost Busters movie?

  1. Taking risks is a crucial part of entrepreneurship
  2. To start a business is a risk
  3. Getting money is often risky (mortgage, vc, whatever)
  4. Worth the risk if you can find an innovative solution to a real problem
  5. Markets don't reward effort but the value they create for others (subjective value)
  6. Over-regulation can kill your business
  7. Negative externalities & trade-offs
  8. Micromanagement's negative impacts
  9. Unintended consequences

Honestly, Ghost Busters is a great example of a work of fiction intended for entertainment purposes, where valuable lessons about business show themselves in ways that are applicable in real life. #startup

@jer %W+hWDy1hRX2sGonU3O48JFZRE9OTMK0vU9JOsDf/JLo=.sha256
Re: %JgYgMCLlF

I did say "next highest virtual node" when I meant "next highest node" Oops.

@jer %JgYgMCLlFHFm6dIPpg+BNIxtiOE995kah7jAiyzRzNs=.sha256

One of the things I like about programming out of my comfort zone, is I get great exposure to new things. Even in your comfort zone, you find new things that make some things that you have to do in various applications over and over, simpler. This week, for me, that was the gb_trees module in Erlang.

So I've been working on a hash ring implementation for a financial trading bot app, and as part of that implementation you've gotta figure out how you want to build the data structure to slot each node shard into the ring. Key things you're looking for:

  • Consistent Hashing -- I want the data to be distributed at a certain location based on how it hashes
  • As time passes, new larger nodes come online, I want to be able to weight them differently, so older nodes that can't take as much load don't get as much ... load :)
  • I want to keep data replicas in different zones
  • I don't want very much (< 0.5%) to be outside of the zone caring for it
  • I need a property that if a node is unavailable, the data will be saved to the next highest virtual node; and if there is no next highest virtual node, wrap to the first virtual node in the ring, thus completing the ring

As it turns out, I can use gb_trees for the heavy lifting in my hash ring code, and it performs really well. So the entire hash ring is about 90 lines of code, and supports adding/removing nodes, getting all nodes, and finding nodes to place data on when given a key. This is somewhat unfortunate because it means I have to add 3 nodes at a time to increase capacity.

I'm still thinking about whether or not its worthwhile to spend more time to build the hash ring a different way, such that I can have 1 ring for all my nodes, while guaranteeing good distribution of data on nodes in different zones with little over/under...

The application it's being built for is using this hashring to store data on nodes for order books of financial assets (crypto coins, forex pairs), and data is read out of nodes by my quantitative siganlling service to perform statistical analysis, so it has to be as fast as possible for the signals to be actionable by the order execution service, such that orders can actually get filled. So far, entire latency through the system (from the time the data comes in through the scraper to the time an order is placed) is 114 milliseconds. The slowest part of this whole pipeline is actually the REST call that I have to make to place the order, at 80 ms avg ttl.

This is totally not fast enough, but the ring isn't my bottleneck, the exchange is; so that's at least a good as I can get right now.

@jer %q9ZNfK17MOkIV1G2katUHtC5f4sWtfM/W2+cpFQxxIs=.sha256
Re: %eB7dnmW87

We've got 3 kids, 15, 9 and 7. I will say this, more photos / videos exist of our 7 year old, than our 9 year old, and more of the 9 year old than the 15 year old. This isn't because technology has gotten better, just that technology from 8 years ago or so is more similar to technology today, and unfortunately for some events of the 15 year old's childhood, we've lost pictures and videos because of changing technology.

My big lesson in my kids' lives: Avoid vendor lock in. This also covers the "neat open source project that I'm sure will catch on" temptation.

One thing is for certain, change. I'd strongly urge just backing up the images on different discs, and even putting copies into Amazon glacier (I'd suggest encrypting them, but it might be too easy to lose keys, so whatever your risk tolerance says). Whatever you do, keep multiple copies. Things get damaged over time, bitrot is a real thing. Clouds aren't forever.

Anyway, congratulations on the new addition to the family, hopefully you don't make mistakes I did by keeping some photos in only one place and on one kind of medium.

@jer %0DIpojEc2AVYtnAeEAV3xc11GhcK3CTn9axf7FfhH90=.sha256
Re: %vIAThZDwi

I'd probably argue against this case, since it shouldn't happen. Generally the way when you're in a properly supervised hierarchy is to handle errors that are reasonable, and let the supervision structure restart unexpected conditions.

For instance, if you're on a single node, not in the cloud, but bare metal you maintain, and the DB lives on the same host as the application reading/writing to it; then it's not reasonable to expect the DB to be down, in this case "let it crash". If you're on a cloud service though, where the DB lives on another node, then it's reasonable to assume that at some point your DB is going to be away, and it may make sense to handle that error in a reasonable way (like trying that request to another db server).

Since we're talking about files on the same system the node software is running, imo this is one of those "let it crash" moments, since under what reasonable conditions could that ever fail? And would its failing be fatal, or recoverable?

@jer %Owl9zW+I5VuIc/1AnHSBNY07AOnQRQFlskt0rwuJyTg=.sha256
Voted [@jer](@uf3kajfhBC2Sd3OysKB5LnBLJ5pwgBhBpXnGEGzgG6A=.ed25519) I did not kno
@jer %/ANqEfWh69tJf21aFM4QJ4C0nVr3clhpXJBVcpJG4Y4=.sha256
Voted [@jer](@uf3kajfhBC2Sd3OysKB5LnBLJ5pwgBhBpXnGEGzgG6A=.ed25519) thanks for sh
@jer %kMCYQ7XVS4EAb4sFheI+6f4SumJgt6NvaQ8k/vjDe/E=.sha256
Re: %FECUCa9v3

@elavoie Sorry this reply comes so late, I've been away from SSB for a while.

One story I left out of my original post was about the land the house I owned in my original post, was claimed by. This is the dispute on land that included the 1/2 acre I bought, in 2008. My lawyer didn't tell me about the dispute, I found out about it afterwards. I recognize that as a privilege that i had no involvement in creating. I no longer own the property, and didn't profit of it; it went to foreclosure. I recognize the sticky situation that particular dispute is.

I also recognize that my brain injury gave me awareness of a skill I needed to develop -- memory. Yes, this is a privilege, and I didn't create it; an 11 year old back then, a neighbour of ours, created it for me, but it's mine to live with, epilepsy and all. That injury caused my epilepsy, and is also the reason I have chronic shoulder dislocations in both arms (compression fractures on both humorous'). That's less of a privilege than an injury, since I've yet to recognize any benefit from it :)

I also recognize the privilege of having been homeless, as it was to me. It was beyond my control at the time it happened, but helped me develop a sense for when people, especially creative people, are burning out. It's given me a skill I can use to help coworkers recognize the state of mind they're in, offer help when possible.

I have tried to turn all the privileges I know I have into good things I do not just for myself, but for my community whenever possible. I'm just tired of being chastised because I'm the villain of the day: A straight white man in the west. So thank you for not going down that road. I appreciate it.


Show whole feed
Join Scuttlebutt now