AI Futures: how artificial intelligence will shape music production

MusicTech magazine called it “software that does your mixing”, and the release accelerated conversations that were already bubbling around automation, creativity and AI’s role in the studio. (Automated mastering tools like LANDR and Cloudbounce have been around for a while, but don’t operate within a DAW, and the user has much less control over the output.) 

“Candidly, we initially thought of this as being a useful feature for people who had less experience doing music production. It’s an assistant — it’s designed to help them get a good sound, at least a starting point. We have some spirited debates: from a fully manual process to a fully automated process, where is the right set point on that range?"

For producers and engineers, it’s the closest thing to the clichéd ‘robots will steal our jobs’ rhetoric that’s been circulating since robots were first imagined. But is it a legitimate concern? As the line continues to blur between tool and collaborator, we asked Bailey if iZotope felt a sense of responsibility as an industry leader around AI and ML tech for engineers and producers, and what it could mean for the future of music production. 

“I get asked this question a lot, and my traditional answer is: as a mix engineer, if you make your living by loading up a session, getting the most generic mix in place and not really applying any of your own creativity and humanity to that and moving on to the next assignment, then I’m sorry, you are going to be replaced by technology. If you offer no spice beyond that, then you are purely solving technical problems, and technology will replace you. But I don’t think there are many people that actually work that way.”

For touring DJs who spend weekends on the road, with limited studio time mid-week, reducing studio admin could be a godsend.

“Even a problem as simple as: you’ve done a recording and opened up your session. The levels are pretty good, there’s not a lot of frequency clashing in the mix — everything kind of generically sounds good. That should be a starting point for creative work, not an ending point,” insists Bailey. “There aren’t too many engineers that I know that wouldn’t love living in that world, where they can just focus on the art part of the job.”

For Bailey, the responsibility falls less on the impact of these tools on creativity, but on the ethics around their misuse for the industry as a whole. Once again, the conversation turns to deepfakes.

“The bleeding edge right now in research in the world of deep learning is primarily focused on content synthesis,” he continues. “Re-creations and deepfakes would be an example of this, where we can synthesise content that had never existed before. There are some pretty interesting ethical questions that come up on both sides of the equation. Are we using people’s data to create models in an ethical way? That’s really important to me.

“On the flip side, are the applications of our algorithms being used in an ethical way? I don’t know if we can guarantee that,” he admits. “If someone used an iZotope tool to create a Miles Davis solo that never existed, is that ethical or not? Is that good for the world or not? I’m not sure. That’s a challenge as a civilization that we have to confront with the power of these tools.”

Deepfakes may be the most obvious cultural question mark over what’s possible with AI and ML music-making in coming years, but there are other implications for creative individuals. For those who are working to tight deadlines, or whose ears have grown tired from hours in the studio or have been on the road all weekend and need to finish a remix, self-doubt can creep in. Being able to reach for a tool that is, technically, always ‘correct’ or as ‘correct’ as it’s been taught to be, could throw up a fascinating creative dichotomy that plays on the idea of the imposter syndrome that many producers feel, be it their first track or 20 years into their career. Getting a second ‘opinion’ from an AI tool to show you’re at least on the right track is a tempting proposition. The algorithm, after all, ‘knows’ everything, and if you’re working in an untreated room, with entry-level speakers and you don’t have decades of engineering experience behind you, it makes sense you’d be tempted to take the AI’s word over your own. And while collaborating with a fully-formed human is about bouncing ideas, spurring each other on and bringing sometimes-contradictory musical influences to the table, it’s also about sharing the burden of putting your art out into the world. Once AI becomes a more competent collaborator, more conceptual questions will need to be asked. For now though, iZotope’s focus is purely on problem-solving.

“When we design the tools, we don’t see it that way,” says Bailey. “Our mission is to provide tools that enable people to be creative. Audio creation and audio production have gone from a place of being a highly, highly technical discipline and through the invention of disruptive technology has become less and less about the technical space and more about the creative problem space, thereby allowing more people to participate [in it]. 

“Audio masking between two audio tracks, that is not a particularly creative problem to have. But asking ‘Should the bass or the vocal be more prominent in the mix during the bridge?’ that’s a creative problem and ultimately the creators should be making those decisions, [not us].”

Is it inevitable then, that over the next few years, every plugin will have some kind of AI functionality? 

“There’s a bit of a split in the market in how plugins are designed,” explains Bailey. “There’s the emulation of analogue hardware, and then there are pure software products like iZotope and FabFilter. I’m aware of companies in both categories that are using deep learning, even companies like Univeral Audio whose whole thing is emulating analogue gear, ML gives us techniques to do that more effectively. My opinion is we’re gonna see more and more of it.”

As companies like iZotope and UA embrace the potential of AI both under the hood and as part of the user experience, other companies are taking a more extreme approach, completely re-thinking what AI and ML could mean for the modern DAW and re-writing the rule book on how music is made using computers. 

Never Before Heard Sounds was founded in 2020 by Yotam Mann and Chris Deaner. ​​“I felt like there was a huge opportunity for things that were not just showing off the power of AI, but using the power of AI and giving it to actual musicians to make new and interesting music,” says Mann. By putting the musician first, and the tool second, NBHS is aiming to re-think how we interact with AI in the studio and on stage. “As musicians, we’re not interested in automatic music creation technology, which is a lot of the direction AI music production seems to be going in — very few people are trying to build tools to augment musicians instead of replace them.” 

As well as collaborating with Holly Herndon on Holly+, they’ve built a website that lets you resynthesise audio from any video on YouTube and convert it into either a choir or a string quartet. Try it yourself here. Remarkably, they’ve also built a prototype piece of hardware that can resynthesise any incoming audio into a model of another sound, in real-time — a world first.  

“The angle we take is not to try and wrap it in magic and say some auto-magical AI creature is going to fix your musical problems for you,” continues Mann, “but to try and make the algorithms as transparent as possible and as useful as possible. That’s why we call our tools instruments. They’re meant to be held and played with, and not apply some kind of magical layer on top of your production.”

NBHS co-founder Chris Deaner, an accomplished drummer for the American band Plus/Minus, adds: “In some ways what we’re doing is a lot less sexy than a tool that does everything for you, which was and is the promise of AI and ML. For us, both being musicians, we’re really interested in the human portion of this.” 

The human portion of modelling in particular, as we discussed in part one of this series, brings with it its own ethical challenges. 

“One risk of AI is that you end up with a song-generating machine that makes some beautiful compositions but it never credits any of the things it used to make it,” says Mann. “So for us, it was really important to showcase the musicians that went into it. We think of these generative models, not as AI entities themselves, but as a conduit between you, the end musician, and the musicians involved in the modelling.”