not having the ids in base64 directly makes the hashtable index faster. It doesn't need to parse the base64, it can just read an int out of the key and use that as the key. Also, I realized that a the current hashtable view clears a timeout and creates a new one every write. (the idea here is to write only when things stop changing, but clearing and creating half a million timeouts actually takes some time.

Adding:

if(timer) return
timer = setTimeout(function () {
  timer = null
  async.write(data)
})

means that index gets built in 2 seconds instead of 7! that's a huge improvement! I also suspect this change will effect flumeview-reduce too.