How I built a musical instrument controlled via hand-tracking running in a browser in 24 hours

Do you know that feeling of overwhelm just before starting a challenging project followed by urge to procastrinate? I do, too. That's a place I found myself in when I nearly gave up the day before TotallyMoney Big Hack.

I've never participated in a Hackaton before. Wasn't sure what to expect. I don't need to give myself even more work, I need a break from coding, I thought. I thought it's going to be lots of effort with little reward. How much, in fact, can you do in 24 hours? Not enough for it to be worthwile, I thought. I couldn't have been more wrong.

Let me start from the beginning. Given my musical background and that I work as a Front End Engineer I decided to build something with Web Audio API. Building a synthesiser from scratch seemed exciting. Initially it was meant to be a one-man job, but soon after I announced my idea, 2 colleagues joined in. Idea evolved to building a synthesiser with a step sequencer.

We started by discussing what approach we want to take, decided to use a combination of TypeScript and JavaScript and a simple 2-dimensional array to hold the state of the step sequencer.

A piece advice for anyone participating in a hackathon: TS can initially slow you down, not a good thing for a project to be built within 24 hours.

I'm at my best when I work alone, but this was a good reminder of how energising, and stimulating working closely together on an exciting project with other devs can be. One of my front-end colleagues at work shared a video inspired by Theremin (musical instrument played without without physical contact) saying that these are his expectations for this project. I'm sure this was partly a joke, but I thought of it as a challenge.

That made me think, surely, in the age of machine learning, it can't be that difficult to track user's hand with a webcam and then translate that position to some sort of an input for the synthesiser?

Well, it appeared that there already was a library for prototyping real-time hand detection in the browser (handtrack.js) making this task pretty straight-forward. Having already built core audio functionality for the step sequencer it was easy to plug it into it, and there you go, you've not only got a step sequencer, but also a theremin that you can play right in your browser!

You can try it here: TotallySynth.
TotallySynth on Github.

Just be careful with volume when using headphones! Also it's pretty buggy - it was built in a day, after all!

Hack Day happened to be one of the most exciting moments of my career as an Engineer recently. It reignited my passion for coding and reminded me of the value of human connection at work. If you're wondering if you should participate in one, just do it. Maybe it won't change your life, but might give you the reason to finally write that first blog post that you've been wanting to write for months.

• Next: Blog post ideas