I recently wrote about how Daniel J. Levitin’s book I Heard There Was a Secret Chord: Music as Medicine explains my visceral reactions to music. In his book, Levitin explores how music is processed in the brain and music’s therapeutic effects, which go beyond simply distracting us from pain. The book contains interesting anecdotes and case studies about various neurological disorders, but the part about music processing is what caught my attention. This article summarizes those processes from a layman’s perspective and explores how what we know about music processing could be applied to machine learning and generative AI.

How music is processed in the brain

The way music is processed in the brain is really interesting. When raw audio signals from your inner ear reach your brain, each individual element of the sound is processed separately before being synthesized into a coherent signal that we understand as a particular sound.

Separately! That means that timbre, tempo, rhythm, pitch, melody, and harmony are all processed independently in different pathways in the brain. There’s a part of your brain that helps tell a trumpet from a vacuum. Only after these disparate processes are complete do we recognize the sound of a piano and Paul McCartney’s voice in the introduction to “Hey Jude.”

How do we know this? Partly through brain scan research, and partly through case studies. When specific parts of patients’ brains are damaged, they can lose specific abilities, such as being able to distinguish different sound sources. One patient couldn’t tell the difference between an orchestra playing a symphony and a thunderstorm.

How that maps to training generative AI models

It quickly occurred to me that this knowledge could be helpful in developing AI systems that could understand music. The applications of AI in music could be incredible. If you could train a large language model to understand music, it could serve as a songwriting partner, for example, by writing down chords as you play them on a guitar and automating the creation and distribution of lead sheets for a musical group. When AI can analyze the various elements of music, it could become a creative partner or music teacher. In the way that we use LLMs now to explain codebases, music students might be able to use LLMs to explore musical structures in symphonies or extract and explain soloing techniques from famous artists.

As it stands, there’s a lot of progress to be made in this area before AI is of any use. Some examples, however, like Spotify’s LLark, show that the concept of combining distinct, specific streams of analysis into a composite representation of an audio recording of music holds great promise. In their article, they show how the exact process that Levitin describes in his book is used to generate training data for their model.

They then use this training data to fine-tune an LLM (LLaMA) based on real audio input generated from OpenAI’s Jukebox music generator model. They feed the original audio into Jukebox, which generates an embedding. They then translate that embedding into a format that LLaMA can understand. This input, along with a text prompt like “What is the key of this song?”, fine-tunes the model to improve its accuracy with the audio inputs. This is similar to the workflow used in classical machine learning.

Jon Flynn references this work in his own attempt to create a music transcription model. Without access to the training weights that Spotify’s research produced, he was unable to get his model to produce usable results, citing a lack of computing resources to reproduce Spotify’s work. Other models, such as MOSS-Music (on GitHub), show results similar to those of Spotify’s research.

Research by Spotify and Flynn focuses on getting AI to understand musical audio sources. This is clearly still a challenging area of AI development. AI does much better when the audio input is converted to a digital representation like MIDI. Jordan Rudess, the keyboardist from the band Dream Theater, recently collaborated with MIT’s Center for Art, Science, and Technology to develop an AI model that could play music in a call-and-response format—in real time. The model they developed can play either monophonic (single-note) solos or chord changes. It does so by interpreting the digital MIDI code of a musician’s live performance and generating MIDI output in response. You can watch Jordan demo his AI on Rick Beato’s YouTube channel.

Combined with what I know about machine learning, Levitin’s book on the neuroscience of music has given me new insight into how AI could be taught to understand music, opening the door to tons of exciting possibilities. I will continue to learn about this area—I’ve got some intriguing application ideas to explore.

References and further reading