AI Futures: how artificial intelligence will change music

Machine learning is an arm of AI that essentially teaches a machine how to learn. It uses ‘training data’ to spot patterns and uses those patterns to build a model based on that data. Deep learning is a subset of machine learning that relies on large artificial neural networks to build these models, and it's behind most of the breakthroughs we see in modern AI. 

Spotify’s Discover Weekly is the most obvious example of this. Apple’s voice assistant Siri is another; its synthesised speech is learned from real-world recordings, and it better recognises your voice over time. (For the purposes of this article, we’ll use ML as a catch-all term for both machine learning and deep learning.)

But it’s not just recommendations that ML can attempt to master. A form of ML has been used since the 1980s to generate music, when composer David Cope trained a computer on Bach’s catalogue in an attempt to beat writer’s block. More recently, artists like Actress and Holly Herndon have been training models and building data sets on their own music, vocals, stems and style, in an attempt to create a virtual collaborator modelled in their own likeness. 

Plugins and music-making software have also begun to adopt ML, with iZotope’s Neutron welcoming a new era for how we produce music in the studio. More recently, Splice, Loopmasters and more are using ML to recommend new samples to improve your track, and allow you to scan their libraries of millions of sounds based on more abstract attributes like harmonic profile and tone. Apps like Endel and AIMI have used stems from collaborators like Richie Hawtin, Grimes, Black Loops and Shanti Celeste to create personalised generative music that never repeats, and never ends.

DJing hasn’t been left untouched by ML, either. VirtualDJ and Algoriddim’s djay software have introduced real-time stem separation powered by AI. DJ software’s AutoMix functions also use ML to understand how a song blends with the next, offering the perfect automated DJ for your next after party. Much like Spotify, Splice and Loopmasters, advanced recommendations and search are becoming more commonplace in all DJ software, while Pioneer DJ’s rekordbox recently introduced an AI-assisted vocal detector, to avoid dreaded vocal clashes. Sensorium Galaxy, a new virtual reality platform, has recruited Carl Cox, Eric Prydz and Charlotte de Witte to perform inside its virtual space, with support from AI DJs trained on hundreds of hours of electronic music. 

The implications of this technology on how we make, perform and listen to music will be wide-ranging and, at times, dramatic. New, unprecedented legal and ethical questions will arise and deepfakes will alter our perception of what’s real. Music-making instruments, DAWs and tools will be completely re-thought and re-designed. Fundamental mixing and production skills will be automated, virtual DJs will master mixing and track selection, and collaboration — or identity theft — will be possible with any artist, dead or alive. 

To explore these issues, we spoke to key artists, developers, start-ups and experts to find out how AI will shape electronic music’s future.

The Music-making industry is about to face a storm of deepfakes. Deepfakes are synthesised media — be it video, audio or image — that use AI to replace one person’s likeness with another’s. There are many examples on YouTube, largely of celebrities or politicians morphing into other celebrities or politicians with staggeringly accurate results. While Forbes proclaimed that “Deepfakes are going to wreak havoc on society” and “we are not prepared”, for music it’s slightly less ominous, but still a concern. 

Where sampling introduced an ethical and legal shitshow from the mid-’80s onwards, deepfakes and ML models are likely to trigger the next wave of debates, controversies and lawsuits throughout the 2020s and beyond. While with sampling, there are tools that can pitch shift, tempo change, warp and manipulate audio, the output is always relative to the original sample. 

Models built on ML, be it of a vocalist, guitarist, drummer or even a full mix, can identify and replicate highly personal idiosyncrasies in that artist or producer’s style; replicating every detail of a lifetime of musical and technical mannerisms and characteristics. When we can’t tell the difference between a real Beyoncé vocal and a fake one, the can of worms is well and truly opened.

“I think this is going to be one of the biggest ethical conversations in music going forward — and we’re so not ready,” says Holly Herndon. She’s been heavily invested in AI and ML for years, releasing albums and projects that use cutting-edge technology. For her 2019 album ‘PROTO’, she created a voice model named SPAWN. This July, she released Holly+, a project that lets anyone upload audio and have it processed and recreated in the Holly+ algorithm’s interpretation of her voice, all based on hours of machine learning. 

“On the last album we made voice models, and we thought it would be really interesting to open that up and let other people play with my voice,” she says, “because we see this as something that’s very much on the horizon.”

How that horizon unfolds is still uncertain. “This could go a number of different ways — it could be a total nightmare scenario where people are making work with your vocal likeness in a way where you’re not really comfortable and you could try and control it and go really DRM-heavy [Digital Rights Management] on it. We decided to go the other way and make the model freely available to everyone.” 

Herndon’s take is not to try and restrict the technology, but to embrace its creative potential.

“It’s almost like trying to stop people from Photoshopping your image,” Herdon says. “The cat’s out of the bag.”

“If tomorrow, we could make a model of Beyoncé’s voice and then release a song credited to ‘Holly Herndon featuring Beyoncé’, we’re probably going to have a lawsuit,” says Mat Dryhurst. He’s a musician, researcher and lecturer at New York University’s Clive Davis Institute of Recorded Music, as well as Herndon’s long-time collaborator. “There’s going to be people who do that shit just for the attention too, which is annoying. It’s way cooler to jump out in front of it and act responsibly, but also celebrate what’s rad about it at the same time.” 

“How cool would it be too if Beyoncé actually embraced it,” adds Herndon playfully, “and when you went to your local karaoke bar you could sing ‘Single Ladies’ as Beyoncé?”

Whether the artist in question welcomes the new technology or not, moral, ethical and legal questions remain.

Professor Joe Bennett, a forensic musicologist at Berklee College of Music told Billboard earlier in 2021, “The sound of the voice itself is not covered by copyright law. The reason you can impersonate someone with deepfake audio is because there are only two protected objects in copyright law – the musical work and the sound recording. The musical work refers to the song – notes, chords and lyrics. And the sound recording protection can only be applied to a specific track. This means deepfake audio is a grey area, as the voice isn’t considered [by law] as a part of the composition.” 

There is, however, legal precedent to sue over a vocal likeness, as Herndon explained. When US car company Ford tried to license a Bette Midler song in 1988 and she refused, they hired a likeness – her own backing singer – to re-sing it in her style and tone. She ended up winning the case for infringement. The case is likely to be significant as the inevitable legal fightback against deepfakes grows. 

While an identical model trained on one singer may be more obvious, it’s when the data used to train the model is less clear that the waters become muddied. 

“People describe this ‘black box’ and it’s really hard to pinpoint why an algorithm will output what it does,” says Cherie Hu, a leading voice in music technology trends, an award-winning journalist and researcher and author of the excellent Water and Music newsletter and community. “It’s a general issue in machine learning, and for music, there are very particular implications for copyright ownership and royalty payments.” 

If you wanted to create a model based on Bicep, Herbie Hancock and Slipknot melodies, who, if anyone, is compensated for your use of those artists’ intellectual property? While sampling can be obscured into oblivion, it’s at least possible to identify certain aspects of a sound. For modelling, there’s no start or endpoint, no clear references and no way to know what music was used in the training. Even if your intentions are to compensate, it’s impossible to quantify what chord progression or melody is attributed to which artist. 

“There is a real concern — unless the publishing rightsholders and the artists themselves gain some fluency over this, we could find ourselves with a problem,” Dryhurst continues. “The analogy we always use is: Google made one of the world’s most powerful companies by scraping the internet with no one’s permission. Google was like, ‘We’re going to index all of it and then sell services on top of the ability to navigate that information’. In this new paradigm of machine learning, the same opportunity exists for creating the largest models. One could imagine a new service, or a new DAW, that ties into one of these mega models, where you can just search for who you want your song to sound like.”