Loop Drop Dev Diary

A few people may know that in addition to working on Patchwork, I also develop some electron based music software for live performance and stuff.

I was inspired by my recent patchwork dev diary to also do one for Loop Drop. This probably won't mean very much to anyone else, but putting it out publicly might help me to feel more productive and stay on top of things (especially while trying to juggle the patchwork grant stuff too).

2017-11-20

Continuing tweaking the new looping features that I added last week. Added quantized loop start.

Discovered that the Ableton Link midi clock isn’t as stable as I’d hoped. It is completely broken when I try to sync with my windows computer, and still a little shonky when used with local Ableton Live. It sounds fine to listen to though, so probably okay for jamming. It seemed to be messing a little with the midi clock sync output from loop drop because my FX pedal tempo was showing a lot of jitter.

I’m trying to be able to record the audio output from my synth that is being driven from Loop Drop back in with the LJ audio included. But Ableton screws up the clip warping when I record it with link enabled, and without I lose the tempo info.

Ideally I want to be able to record multi-track audio from inside loop drop. What about adding an “input” global controller? Select the input device and channels you want to record and then they are included in the recordings made from Loop Drop. As far as I know, Web Audio has all the pieces to do this!

2017-11-21

explored adding multi-channel recording support to loop drop

It is possible to get a list of devices using navigator.mediaDevices.enumerateDevices()
you can then choose one of these devices by passing the deviceId constraint to getUserMedia.

However, you can only get the first 2 channels of a device because getUserMedia only allows stereo inputs

So looks like my only option is to do it outside of web audio. This is okay for recording, but unfortunately won’t let us to bring in channels for realtime processing.

First explored shelling out to sox for recording, but it also is pretty bad at multi-channel and multi-device stuff. Also a bit old and bloated for my purposes.

Then I discovered naudiodon a native-addon which allows you to access streams from audio hardware using port audio. It supports accessing any number of channels. Only trouble is that you can’t access channel offsets (as far as I can tell), so to capture channel 7, you also need to capture 1 - 6. Then that will probably involve unmuxing the channels out of the stream.

Also there seems to be an issue with building on macOS right now where you still have to manually brew install portaudio even though portaudio is embedded.

Need to investigate the simplest way to access individual channel streams.

But it is time for lunch!

Well the afternoon is getting on. I said I was going to play music every day this week, and I haven’t yet. Better get on to that! Too bad if I can’t quite yet record the output.

Discovered while jamming that there is definitely something weird going on with the midi clock output.

Also a bunch of sounds are sounding a bit different since messing with the envelopes. But it seems to be possible to fix just by tweaking the params a bit. There are a few issues. First one seems to be that attacks are just faster now (for the same value), and second is that releases behave differently now. Also no attack is disabling the attack rather than the old min 0.005s default.

When I added a param to a glide time, the light on my launch control didn’t reflect that there was a param added (but the knob still worked).

Experiencing a bunch of glitching whenever I interact. But it goes away when I start the profiler. Maybe memory leaks again?

Memory heap dump isn’t working, fails with some error when building the post order index. However the timeline profiler shows the memory heap only growing for each setup opened and closed, and never getting smaller. Definitely some memory leak problems again! That could be causing the garbage collector to be getting jumpy and maybe that is what is causing the glitches whenever knobs are touched? HMMMM!

At least I managed to get a wee bit of jamming in. Looks like the new looping stuff is working pretty well.

Looks like tomorrow I have some memory leaks to get to the bottom of, as well as the midi glitching to solve. Maybe the midi is caused by the performance.now() getting out of sync with audioContext.currentTime?

That and implementing some kind of track recording!!?? Tall order for a Wednesday 1 week before a gig! Gerp.

2017-11-22

Alright, here goes! Straight into memory leak debugging first thing in the morning.

I was thinking about a way to test for this. Maybe some kind of automated test that opens a setup and then closes it again measuring the memory usage before and after. But I’m not in the mood for that today. Time to do it the old fashioned way.

Found out that if you call gc() immediately after unreferencing the objects, they do not get garbage collected. But if you call it inside of a requestIdleCallback, they do! This was confusing me as the heap size was not dropping down after closing setups, but it was when measuring with heap snapshots. My initial test setup was not memory leaking, but I did find some that are. Time to start eliminating bits until it stops.

Strange, while the heap snapshotter throws an error when you have a setup loaded, it does not when you close them. UNLESS it is memory leaking, so while it can’t help me find the the leak, it does tell me if it is leaking. Better than nothing.

Found one actual leak (was missing a destroy on the inner chunk for the new mono-synth). However, there is one setup that will always leak if opened first, but if it is opened after another setup, then it doesn’t leak!! WUT!

I’ve left the “gc() purge after closing” enabled. Will see what happens when this is used with a jam. I removed it in the past because it was causing issues with responsiveness when transitioning between songs. But I think this was because my pre-schedule time was too high. Now preschedule is lastGCDuration * 2.

Will have a jam a little bit later with the memory allocation profiler running and see if there is anything that is peaking a bit hot for memory usage. Still need to figure out why the audio goes glitchy when adjusting knobs (but not when javascript profiler is running).

Back to multi-input / output

Just realised that while Web Audio can’t do multi-channel input, it should be able to do multi-channel output. Gonna try and get a proof-of-concept output channel switcher working. Also would be nice to get the input recorder working from yesterday. Looks like the mac bug has been fixed in that module now!

Oh nice! audioContext.destination.maxChannelCount is 10 when connected to my K-Mix.

Made a cute little MultiChannelOutput module that allows you to set the outputs and channel numbers and it will automatically handle all the routing magic and channel counts for you! Next step is to build a user interface on top. But for now, I’m just gonna set it manually!

And yay, it WORKS!!

_But also found an issue with the clap generator (on musical score) NaNing out the sound. Heard there was an issue with filters ... no fix yet.

So further work here would be the ability to select the outputs, and some kind of refresh when the output device changes

Okay, nearly ready for today’s jam. Let’s see if we can quickly whack out a multi-channel recorder!

Tried out the latest version of naudiodon that wasn’t working without portaudio installed on my mac, but unfortunately there is still a problem (but it’s different). Oh well, won’t stop me from working on this!

Been trying to get external recording working in Loop Drop with the segmented wave writer, but no data is flowing. Strange. Just tried the example and it works fine (pipe to to fs.createWriteStream). Wonder what is different about this way? Oh I’m missing .start(). Still occasionally getting “input overflow” error.

It is using a very small buffer size, can’t figure out how to increase it yet. Doesn’t look like you can, gonna try stream-chunker. That works!

Okay so tried recording multiple channels to one wave file but it’s not behaving quite right. The stream works fine, but maybe something is lazy in SegmentedWaveWriter. Gonna try uninterleaving the stream, and splitting into multiple files.

Trying to figure out an efficient way to deinterleave the stream and recombine into the correct channel mappings and discard unused. TO MANY MATH FOR MY BRAIN RIGHT NOW!

I’M DONE FOR THE DAY! Didn't manage to get a jam in never mind. Some good research today.

Think tomorrow I should try setting up a worker thread to receive the buffers and then do the de-interleaving of the channels in the worker and post back buffered up chunks.

2017-11-23

TL;DR: I spent the last two days adding the ability for Loop Drop to record external outputs as part of the session (so I can include my external synths and FX), and the ability to change what channel Loop Drop outputs on so that I can route it correctly in my mixer!

Overnight, naudiodon was updated to fix the mac bug. So that’s nice!

Gonna have a go at that worker based deinterleaver now. Thinking that you give it a channel map (with channelCount and bitrate info) and desired chunk size, and then it streams back chunks for the different channel pairs and discards ones you don’t want.

Whipped up the worker. Need to pipe some data into it and see if it works! You give it {bitDepth, channelCount, map, samplesPerOutputChunk} and then post in the data as typed arrays matching the bitDepth. It then gives you back buffered TypedArrays with the data mapped as per map.

Okay, so it’s working. But after a few seconds, naudiodon is throwing “portAudio status - input overflow” and stopping. I’m guessing this is because it does not have an internal buffer queue, and if I can’t read it fast enough, it stalls. No idea how to fix this :(

Can reliably reproduce this issue in the tests now by adding a sleep every so often. Filed an issue, hopefully it can be resolved!

But I’ve spent too much time on this multi-channel recording for now. The multi-channel output was actually more important (and I got that working) as I needed to be able to output to the k-mix on a channel that wasn’t 1+2 (that is where the mic input is).

Time to stash those changes for another day, and get on with this mythical jam I was gonna have…

—

So, lunch has got me thinking that the web audio input (that only does 2 channels) wouldn’t be completely useless. There are a lot of times where just the first 2 channels would be all you’d need. I have a weird setup because I also want to be able to run a mic input in (for live streaming) and this needs to go into channel 1 or 2. But there are also other hacks like a second usb soundcard that is capturing aux inputs out of the k-mix. HACKY-AF but it would totally work! This could well end up being the simplest option.

But should I do it now? Hmm… i’ll give it 1 hour and see how far I can get.


30 mins down. This seems like a good idea! Much easier to integrate with the existing code. Got the recorder part implemented. Just need to add the interface for adding the inputs. Inputs will handle the getUserMedia and mapping to AudioNode themselves and then be collected into the SessionRecorder. It would probably be better if all nodes were recorded with a single ScriptProcessorNode, but for now this should work fine.

Ahh.. okay so I went a little too far down the rabbit hole here. Another hour later… but whatever. It is working! I have a selector that lets me choose what to record, and it is included in the session recorder. HOWEVER, the clips are all appended to the end which isn’t particularly ideal. Need to add multi-track playback to the timeline view.

Maybe just quickly I’ll hack it so that it creates multiple tracks for the external recordings.

Yup, only 20 mins later and I have multi-track recording playback working. That’s something at least. The interface doesn’t yet show it, and if you edit the clips, everything will get completely out of whack. But it is good enough for now!! WOOOP!

Too bad the day is over. Never mind, maybe have a headphone jam this evening.

The outputs and inputs are working! And so is the recording. However the recorded audio sounds funny. Seems like echo cancellation is the cause. Looks like I need to call getUserMedia with {echoCancellation: false} because it default to on. That also explains why the samples I recorded back in the day sounded so funny.

Now let’s see if it is in sync! It’s not :( It’s also really loud compared to the LJ output. Probably gonna need to put some kind of volume control on the tracks. For now, we can put a default 0.6 or something on it. But what to do about the sync? Is there a way to detect the delay?

Oh wait, I hadn’t actually disabled the audioCancellation and that was adding some latency. Needed to add a {exact: false} to it for some reason. And now the latency is much better. Oh adding an offset to the recorded clip of 1 / sampleRate * 256 seemed to magically pull it perfectly in line.

AND IT IS ALL WORKING. I JUST JAMMEDED AND PREROLL RECORDED EVERYTHING INCLUDING MY EXTERNAL SYNTHS!!! AWESOME!! Now let’s home there aren’t any more surprises and I can get on with actually preparing the music for my set instead of working on the software for my gig.

I suppose if I record anything good, I can figure out how to export it then! Should update my ableton live export to include secondary tracks. That’s probably more important than getting the loop drop editor working.

Sigh, if only getUserMedia could get more than 2 channels.... then Loop Drop would be a full on DAW!

2017-11-24

Last night, in my sleep, I figured out a few more perf tweaks I can make. I made some tweaks to doubleBind (which is used for just about all of the file -> interface links) the other day that means I can get rid of the direct feedback. This should speed things up a bit (no need to diff the changes twice for every change). When I went to implement it and profiled it, I also noticed that I had left the throttle for templateSlot updates off. It was generating updates (that take 5ms each) every 6 ms. WHAT. TOO MANY. Enabled the throttle and param updates are silky smooth again.

Another thing that happened while I was sleeping last night is that naudiodon supposedly got its ‘overflow’ bug fixed. I’m probably gonna ignore this for now as I got recording inputs working well enough for my current pressing needs. But worth revisiting this in the future!

Found an input bug where opening and closing other setups stops the qwerty input from working. But can’t reproduce. May have been caused by switching in and out of profiler (happened twice).

——

JAMMING YO! Discovered that something is horribly wrong with midi sync over time. Should investigate this! Oh wait, it looks like when you remove a midi clock sync, it doesn’t destroy it.

Also the recording editor wouldn’t let me play past the 8 min mark. STRANGE. But once I deleted earlier clip, it worked.

Just remembered something else I was thinking about last night. The recording editor should have busses. Kind of like Final Cut Pro X roles. When recorded, they they are tagged with their input number and a bus is created. You should be able to edit the volume of a bus, or export it individually.

—

Looks like the reason that the midi clock stops working perfectly after a while is because performance.now() (which is used by midi) and audioContext.currentTime drift slowly out of sync. I was measuring the initial offset and then calculating the midi clock based off that, however eventually the offset becomes greater than the cycle time and it starts jittering. Interestingly enough, the two clocks do not drift (at least over the period I observed) when I’m using my Macbook’s internal soundcard, but when I use my K-Mix it drifts about 5ms every minute.

Now I have to figure out a reliable way to generate a clock that matches the audioContext but isn’t bound to it.

I have set up a ScriptProcessorNode that checks the offset between the clocks every so often and calculates the non-jittered offset in ms. This seems to be enough to stop the midi clocks rattling on my devices.

However I have noticed that the midi messages are getting throttled when Loop Drop isn’t visible on the screen (app backgrounding I suspect).

gah, just spent the last hour and a half messing with speaker positions…. whoops.

(continued)

Back into some code now. Tried to copy over my new looping from the Launchpad MK2 to the mini but the loop hold isn’t working. It is just falling back to “loop last loopLength”. Oh that’s because it is broken on MK2 as well!

Hmm, I think that the quantized start loop should also end the loop too. It should be a “record next loopLength”. Okay, let’s see if we can rewrite this in a way that avoids so much duplication.

Nice got it working. Time to check if Launchpad MK2 still works. But thinking a lot about how I can cut down on the duplication between the controllers. Currently, the Launchpads and Push are pretty much the same minus some light differences (oh and the push does some extra weird stuff too). But still….

Okay it is working great! Let’s push this!

So after a nice little wee jam, I discovered that the recording isn’t always in sync. Oh well, at least it is being recorded. We can get to the bottom of that later. Will be interesting to see if using a single ScriptProcessorNode for recording all channels might give better sync or if it is somewhere more hardware level

And there is definitely a problem with playback in long multitrack files. It seems to take longer to start the further out in the file you are (but not always).

Been thinking that it would be nice to be able to set default values for the mixer params instead of always 0. Would be easy for external chunks, but embedded ones don’t expose a default value. Maybe the mixer itself should show the values and you can set them there? Be cool if it showed you the current knob position too!

—

Awesome, fixed the bug in recording playback. It was a silly copy-paste bug. Also, just rewrote the session recorder so that it only uses a single ScriptProcessorNode. We’ll find out next time I jam how good the sync is now!

Pretty good day today. Spent quite a bit of time working on my set for next Wednesday. Only one song in for planning/setup/tweaking/practice, but the first is always the hardest….