12. Machine listening

This week is about machine listening: how sound becomes structured information that a computer can work with. It works entirely within the digital representation of the four levels of description, and the most useful question to carry through the chapter is how far up the remaining levels a machine can be pushed from there. It sits where music perception, signal processing, and machine learning meet. We follow the whole signal→symbol pipeline: capturing audio and cleaning it up, extracting features (spectral, timbral, harmonic, temporal), finding segments and events, and classifying and modelling how sound changes over time. Machine listening is the foundation for the more focused field of music information retrieval (MIR) and for today’s musical AI.

The examples below use Python code with the librosa audio library, written as a Jupyter Notebook. You are not expected to learn these for this course, but it helps to know what they are:

What machine listening is¶

Machine listening, sometimes called computer audition, is the practice of turning sound—music, speech, or environmental audio—into structured, usable information. It builds on digital signal processing, but with a particular focus: capturing reliable audio, extracting features that reflect how we perceive sound, detecting events and boundaries, mapping patterns to labels, modelling how sound evolves over time, combining cues into higher-level interpretations, and presenting the results in a form people or other systems can use. Müller (2021) and Lerch (2022) are the two standard textbook treatments of this pipeline.

Audio input and preprocessing¶

Whether audio arrives from microphones, interfaces, or files, reliable analysis begins with some basic digital signal processing: resampling, DC removal, level normalisation, denoising, and de-clicking. These steps raise the signal-to-noise ratio, remove systematic artefacts such as low-frequency hum, and make features comparable across recordings, so the later stages behave predictably.

Once the audio is cleaned up, it can be the source for further analysis. A machine does not need to visualise sound to interpret it, but for the examples that follow, it helps to remember that the machine also starts with a waveform. As we have seen in earlier weeks, one of the basic ways to begin any audio analysis is to convert from the temporal domain (the waveform) to the spectral domain (for example, creating a spectrogram using FFT). The running example through this chapter is the excerpt of John Coltrane’s My Favorite Things that you met in time and rhythm.

Feature extraction¶

Feature extraction converts raw audio (waveforms or spectrograms) into compact descriptors that better reflect perceptual and musical structure. Features are commonly grouped by abstraction level:

• Low-level descriptors are useful for timbre and energy tracking. They are typically short-term and include features such as spectral centroid (the spectrum’s centre of gravity, heard as brightness), bandwidth, rolloff, spectral flatness, RMS, and zero-crossing rate.
• Mid-level descriptors are useful for key, chord, and melodic analysis. They focus on harmonic/tonal relationships and include chromagram, constant-Q transforms (CQT), F0/pitch contours, and harmonic-percussive decomposition.
• High-level descriptors are useful for rhythm and structure analysis. They include temporal developments such as onset strength, tempogram, beat tracking, and segment boundaries.

The names are worth skimming rather than memorising: each is a single number or curve computed from the audio, and the ones used later in this chapter are explained where they appear. Some of these work in the temporal domain (waveforms), others in the spectral domain (spectrograms), and some combine both. Extracting musical features from audio is not straightforward. As an example, let us look at a chromagram, which estimates the pitch classes in a recording (the same twelvefold reduction used analytically in harmony and melody):

From the chromagram, we see that there is a lot of chroma energy for E. This is also what the machine finds when calculating the average:

Let us then turn to another feature, trying to estimate the tempo of the song. We can begin by making a tempogram, which estimates prominent tempo profiles over time:

From this plot, it is not so easy to see what the estimated tempo is. However, if we ask the computer to smooth the signal and average across bands, we get a clearer result:

Notice how the instantaneous tempo fluctuates quite a lot. This is common with real-world signals and is one of the reasons we typically average (“smooth”) the signal to find a result. Here the estimated tempo of 161 BPM is quite close to what a human would guess.

Source separation¶

Source separation means isolating individual instruments or voices from a mixed audio signal. Human listeners do this constantly: recall auditory scene analysis from the psychoacoustics chapter, the process by which the auditory system organises a mixture of sound into separate streams. Machines face the same problem, and it is still not fully solved. A simple example is shown below, where the signal is split into a “harmonic” and a “percussive” component:

Traditional signal-processing methods such as this are mature and well understood. Deep-learning approaches, which learn the sound of voices, drums, or bass from large collections of recordings, often do better in difficult cases, such as separating many overlapping sources, but rule-based methods remain useful when you need real-time results or are working with limited data and compute.

The most capable systems today can split a finished mix into stems, separate tracks for vocals, drums, bass, and the remaining instruments:

Image source: from Manilow et al. (2020).

A range of source separation tools is available, each with its own trade-offs:

• Commercial online tools (e.g., moises.ai, lalal.ai): fast, cloud-based processing with intuitive interfaces. These services usually give high-quality separation for common instruments in Western popular music, which makes them accessible to non-experts. Control over the algorithms is limited, and full functionality usually requires a paid subscription.
• Open source tools (e.g., Spleeter by Deezer, Demucs): free and highly customisable. Users can adjust parameters, retrain models for specific instruments or genres, and integrate separation into research workflows. They require command-line or programming experience, and processing speed and quality depend on local hardware.

Which tool fits best depends on your technical background, the complexity of the task, and whether you value ease of use or customisation. Separated stems are useful far beyond remixing. Isolating a drum or bass track makes onset detection and timing analysis much easier, as noted in time and rhythm.

Segmentation and event detection¶

Segmentation and event detection are about locating boundaries and discrete events such as onsets, beats, or section changes. This is the machine’s version of the grouping problem Bregman posed in psychoacoustics: deciding which parts of a continuous signal belong together. The auditory system solves it before you notice there was a problem; an algorithm has to be told what counts as a boundary. A traditional method looks for repeating patterns with a self-similarity matrix. The signal is compared with itself, and recurring patterns become visible:

The self-similarity matrix shows the pairwise similarity between frames. Repeating sections appear as off-diagonal parallel stripes, and boundaries appear where similarity across the diagonal drops. We can then ask the computer to (A) detect boundary times (change points) and (B) group frames into part labels (verse/chorus/bridge, or repeated segments). Combining local novelty (for transitions) with global clustering (for part grouping) is the standard practical route from a self-similarity matrix to musical parts and transitions.

It is further possible to compare the segments to evaluate the musical form. However, this requires more code and interpretation, so we will not consider that further here.

Worked example: what can we now say about this track?¶

Each step above produced one number or one picture. On their own they are just outputs. The point of machine listening is what happens when you put them together and ask a musical question of them.

Reading the output critically¶

A summary like this is only worth as much as your willingness to check it against your ears. Play the excerpt again and ask three questions.

Is the key estimate meaningful? The chromagram is confident about E, and the key profile agrees on an E centre. That much is a fair description. But the profiles were derived from Western common-practice tonality, and they force a choice between exactly two answers: major or minor. Coltrane’s treatment of this tune is modal and moves between minor and major over the same centre, so a single label averages away a contrast that is central to how the performance works. A tool that can only choose between two categories will always choose between them, however badly they fit.

Is the tempo the tempo? The estimate is a plausible number, but “the tempo” of a piece is not a single physical quantity. Beat-tracking algorithms routinely lock onto a subdivision or a grouping rather than the level a listener would tap, which is the notorious octave error of doubling or halving the true rate. The waltz metre of this tune makes that especially likely. Tap along yourself and compare.

Are these the sections? No. The segmentation returned twenty boundaries because the code asked for twenty (librosa.segment.agglomerative(chroma, 20)). Change that number and you get a different musical form. The algorithm finds points where the chroma texture changes; whether those points are musically meaningful boundaries is a judgement it never makes.

This is the honest summary of what machine listening gives a musicologist: fast, consistent, reproducible descriptions of acoustic properties, over more music than anyone could listen to. What it does not give is musical interpretation. In the terms of the four levels of description, it is very good at the physical signal and the digital representation, it approximates perception where someone has built a perceptual model into the features, and it does not reach interpretation at all. The features are evidence; the argument is still yours.

Music information retrieval¶

The last example already moved us from general machine listening into music information retrieval (MIR), a more specialised form of machine listening aimed at pulling structured, useful information out of musical audio and related data. MIR draws on signal processing, machine learning, music theory, and human-centred evaluation to support tasks such as metadata extraction, music search, analysis, learning, and interactive applications.

MIR tasks¶

• Classification and recognition includes identifying instruments, genres, or whether a recording contains speech or music. When working with large music collections, this means tools that can tag files, find similar pieces, or detect instrumentation in recordings. Methods range from simple spectral/timbral feature matching to modern neural classifiers. Common pitfalls are dataset bias, noisy recordings, and confusion between timbrally similar instruments.

• Recommendation systems combine audio-derived features (timbre, tempo) with user behaviour to suggest music. Content-based recommenders use the audio descriptors themselves. Collaborative methods exploit listening patterns across users. Many modern systems are hybrid, and they help discover new music but are prone to bias and raise privacy concerns.

• Automatic transcription systems convert audio into symbolic scores (MIDI, MusicXML). Monophonic pitch tracking is quite stable these days, but full polyphonic, multi-instrument transcription remains a challenge. Techniques include spectral analysis with onset/pitch heuristics, probabilistic sequence models, and end-to-end deep networks.

• Segmentation and structure analysis help find meaningful regions (intro, verse, chorus) or discrete events (onsets, section boundaries) in audio files. Approaches use self-similarity and novelty detection or trained boundary detectors. These tools are useful for form analysis, remixing, and practice. However, “meaningful” boundaries are often subjective, so evaluation uses tolerance windows and human judgement.

• Question-answering systems let you query audio with natural language (e.g., “Find tracks with a trumpet solo” or “show passages in major mode with fast tempo”). They combine audio tagging, metadata, and lyrics to match queries to musical passages. The hard part is mapping everyday language to reliable musical labels.

• Mood, emotion, and other higher-level feature extraction are active research topics in the MIR community. These estimate human responses to music, which is where MIR crosses paths with music psychology research. This is challenging, given large individual differences and the impact of cultural background and context.

Music data¶

Data is at the core of any information retrieval system. Traditionally, MIR tasks worked mainly with symbolic data, in the form of MIDI or MusicXML files. Over the last decades, most research has shifted to audio files, which can be seen as a form of subsymbolic data: they do not deal with “symbols” but with the raw audio itself.

Additional information can be found in metadata, including contextual information such as artist or album details. Paradata can tell us how the data was recorded or analysed, while user data can reveal personal information about how people access and use it, such as listening patterns and preferences.

For researchers, one major challenge is getting access to diverse music data while respecting legal and ethical constraints. Copyright, performer and composer rights, licensing terms, and privacy (for user or performer data) all restrict what can be downloaded, redistributed, or used for training and publication. These constraints affect dataset size, representativeness, and the ability to share reproducible code and trained models.

To work around some of this, researchers often turn to openly licensed resources and well-documented research datasets, and publish the licence and provenance of each item used. Another option is to rely on metadata or derived representations, such as spectrograms, when raw audio cannot be redistributed, which reduces legal risk and improves reproducibility.

“Multimodal” MIR¶

Multimodal MIR combines audio with other kinds of data: lyrics can disambiguate mood, scores and MIDI give exact pitches and timings, and video shows performer gestures. The combination also makes systems more robust, since gaps in a noisy recording can sometimes be filled from an aligned score or lyric line.

Working across data types raises two recurring problems. The first is temporal alignment, since audio, scores, lyric timestamps, and video frames run on different clocks, and small timing errors can break a task such as matching lyrics to sound. The second is comparison, which many systems solve with embeddings: lists of numbers produced by a trained model that summarise a clip, a lyric line, or an image so that similar items get similar numbers.

Concrete examples include score-informed source separation, which needs tight audio–MIDI alignment; lyric-aware search, which needs reliable text timestamps; music-video analysis such as visual beat tracking, which depends on accurate audio–video synchronisation; and recommendation systems that fuse audio with listening logs, which raise privacy and fairness questions.

Artificial intelligence¶

Artificial Intelligence (AI) is commonly used to describe machine systems that perform tasks that normally require human cognitive abilities, such as perception, learning, reasoning, planning, and natural-language understanding. In music, this includes systems that can compose, perform, and analyse music.

Artificial neural networks¶

Most learning-based systems in this chapter are built from artificial neural networks (ANNs). The brain explains where the metaphor comes from and how far it stretches; here is the mechanism. A network is a set of simple units arranged in layers. Each unit multiplies its inputs by a set of weights, adds them up, passes the total through a non-linear function, and hands the result to the next layer.

A simple artificial neural network with an input layer, one hidden layer, and an output layer.

Training means showing the network many examples, measuring how wrong its output is, and nudging every weight a little in the direction that would have made it less wrong. Repeat this a few million times and the weights settle into something that generalises, or at least that works on data resembling what it was shown. Deep learning simply means stacking many layers, which lets early layers pick up local detail (a spectral edge, a transient) and later layers combine those into something more abstract (a timbre, a chord, a genre label).

Two consequences matter for the rest of this chapter. The features are no longer chosen by you but discovered by the network, which is what makes these systems powerful and also what makes them hard to interrogate. And whatever the training data contained is what the network learned, which is why the ethics section below is about datasets as much as about models.

Rule-based vs learning-based systems¶

Although learning-based systems get most of the attention these days, it helps to distinguish two broad types of AI system: rule-based and learning-based. There are also hybrid approaches that mix the two.

Rule-based systems rely on deterministic signal-processing heuristics—peak picking, thresholding, spectral rules, and simple decision logic—to make decisions. They are compact, easy to interpret and tune, and well suited to low-latency, resource-constrained tasks such as onset detection or beat tracking. Their disadvantages are brittleness to noise and domain shifts (data unlike what they were tuned on), extensive hand-tuning, and limited ability to capture complex, data-driven patterns.

Learning-based approaches range from classical statistical models trained on hand-crafted features to modern deep networks that discover their own features (see the model families below). When labelled examples are scarce, models are often first trained on large amounts of unlabelled audio, for example by learning to judge whether two clips come from the same recording, a strategy called contrastive pretraining, and only then adjusted for the task at hand. These methods typically offer higher accuracy and adaptivity, but require more labelled data or compute, can be harder to interpret, and demand careful validation against domain mismatch and dataset bias.

Hybrid approaches combine signal-processing knowledge with learned components, for example letting a handcrafted onset detector propose events that a learned model refines, or enforcing musical constraints such as tempo and harmonicity on machine-learning outputs. A practical workflow is to start with a simple rule-based baseline and introduce learned components only where they earn their added complexity.

Models¶

You do not need to be able to build these, but you will meet their names in the literature, and it helps to know roughly what each one is good at. Broadly: sequence models (HMMs, RNNs) walk through a signal step by step and are good at things that unfold in order, such as notes or chords; convolutional models look at a spectrogram much as an image, and are good at texture and timbre; transformers can relate any moment to any other, which suits long-range musical structure but costs a lot of computation.

Realtime vs offline¶

Machine listening systems can operate in different time modes, and the choice strongly affects algorithm design, latency, and evaluation.

• Realtime (online) — the system must answer while the music is still playing, as in a live accompaniment system or an interactive installation. It can only use audio that has already arrived, and it has a few tens of milliseconds to decide. That rules out anything that needs to see the whole piece, and usually means simpler, lighter methods.

• Offline (batch) — the system gets the whole recording and can take as long as it likes. It can look forwards as well as backwards, revise earlier decisions in light of later ones, and run expensive smoothing. This is the setting for transcription, structural analysis, and dataset annotation, and it is where accuracy is highest.

• Hybrid — run a fast online pass for immediate feedback and a slower offline pass afterwards to correct it.

The trade-off is unavoidable. Musical structure is easiest to identify once you have heard how it turned out, and a live system never has that luxury.

Design decisions should follow the application. Live interaction needs minimal, predictable latency and causal algorithms, whereas analysis and annotation can exploit full-context offline methods for higher fidelity.

Examples¶

There are many practical examples of machine listening and MIR in both research and industry. Below are curated open‑source projects followed by some commercial examples.

Listening to a room¶

The examples above analyse music. But the machine-listening pipeline of this chapter—waveform → framing → features → decision—does not care whether the sound is a saxophone solo or a café. Here we point it at a room instead, using ambiscape, a small toolbox for long-duration soundscape analysis (pip install ambiscape).

One idea makes the shift worth the trouble. Music information retrieval was built to attend to the foreground—the notes, the beat—and to treat everything else as “background noise” to filter away. A soundscape inverts this. Its subject is the background: the keynote hum, the diffuse field, the slow envelope of the day. Analysing a room is often music analysis with the attention turned around.

We first synthesise a short “room”: a low-frequency mechanical rumble (think of a fan or ventilation) with a few broadband events (a door, footsteps) over a faint floor.

The room’s spectral centre of gravity sits low and its flatness is small—a tonal rumble, not a broadband hiss—and the source decomposition names it: this room is mechanical, with the great majority of its energy below 250 Hz. Run the same analysis on a real 27-minute train ride and the verdict is starker still, with about 93% of the energy below 250 Hz, an “enormous machine that happens to be carrying people,” over which the announcements and conversations are brief, marginal figures.

The same tools carry the same ethical weight as any machine listening. Because a soundscape is often recorded in an inhabited, private space, ambiscape can export a non-identifying feature stream (a low-rate table of level and spectral descriptors, below the timescale of intelligible speech). It also includes a voice-activity gate to check an excerpt before it is shared, which is a concrete form of the principle from the ethics section below: publish features, not recordings.

Hearing direction: ambisonic recordings¶

A single microphone captures how much sound there is, but not where it comes from. An ambisonic recording, introduced in electroacoustics, stores the sound field itself in four channels known as B-format: an omnidirectional channel W capturing overall pressure, and three figure-of-eight channels X, Y, and Z capturing the pressure gradient along the front–back, left–right, and up–down axes. Together they describe, at every instant, how much sound there is and which way it is going. AmbiX is the channel ordering and normalisation convention most software now expects.

For machine listening, this offers an analytical handle that no mono recording can. Combining the omnidirectional channel with the three gradient channels yields, frame by frame, an estimate of the direction energy is flowing from, and of how strongly that estimate agrees with itself. A single dominant source produces a consistent direction; a reverberant field produces directions that point everywhere at once. The resulting diffuseness measure separates a room with one talker in it from a room full of murmur, even when the two have identical levels and spectra. For soundscape analysis, direction and diffuseness sit naturally alongside the level and spectral descriptors used above, telling you whether a room’s sound has a focus or surrounds the listener from everywhere.

Listening to a collection¶

The same pipeline scales once more. We have pointed it at a single piece and then at a room; the third step is to point it at a whole library and ask how the tracks relate to each other. musiscape does that (pip install musiscape), taking a folder tree where each subfolder is an album and producing visualisations you can compare at a glance: how albums differ, which tracks resemble which, how internally consistent each album is, and where the outliers sit. Its documentation has an illustrated gallery of the styles, which is the quickest way to see whether it would answer a question you have.

It is worth noticing what changes as you move up these three scales, because it is not the method. At every scale the pipeline is still waveform, framing, features, decision. What changes is the question you can sensibly ask of the output. From one piece you can ask about notes and phrases; from a room, about states and rhythms measured in minutes; from a collection, about similarity and grouping. Try asking a collection-scale question of a single recording and you will get an answer, and it will not mean anything.

musiscape and ambiscape are siblings and share machinery, including the circular statistics (statistics for values that wrap around, such as pitch classes or clock times) and the Schaeffer-derived typology you met earlier. That is deliberate, since a tonal centre is a tonal centre whether you are describing an album or an afternoon in a courtyard.

Machine vs human listening¶

Human (music) psychology and machine listening/technology form a mutually reinforcing loop. Psychological knowledge constrains and guides algorithm design, while technology supplies tools, data, and interventions that extend what we can measure, model, and apply. Key points and practical implications:

How music psychology informs technology¶

Knowledge of how people hear guides what machines are built to measure, from perceptually scaled features to evaluation by listening tests. The details are collected below for those who want them.

How technology supports music psychology¶

Technology repays the favour with better stimuli, larger datasets, and new instruments for measuring listeners. Again, the details sit below.

Some considerations¶

Ethical and cultural considerations¶

Make stimuli and datasets ecologically valid. Include diverse musical styles, real recordings (not only synthetic stimuli), and contextual cues so that models generalise beyond narrow laboratory conditions. Design experiments and datasets to avoid overfitting to artefacts of studio recordings or annotation conventions.

Document provenance, consent, and licensing, and keep humans in the loop. Record dataset licences, participant consent, and known limitations for reproducibility and ethics. Interactive workflows (active learning, human‑in‑the‑loop tuning, and iterative evaluation) often produce more useful and trustworthy systems than fully automatic pipelines.

Perception and meaning are culturally conditioned, so models trained on narrow corpora may misrepresent or marginalise musical practices.

Privacy and consent matter when using user logs or physiological data; prefer anonymised, consented datasets and report provenance.

Be cautious about medical or therapeutic claims; collaborate with domain experts and validate clinically.

Bridging music psychology and technology yields systems that are both effective and meaningful. Psychology grounds what we should measure and predict, and technology scales, implements, and tests those hypotheses in real-world systems.

Generative music AI¶

Everything in this chapter so far has been about analysis, extracting information from music that already exists. Generative music AI turns the pipeline around and produces new music Briot et al., 2020. The two directions are closely related, since a system can only generate convincingly in a style it has learned to represent, and generative models reuse the model families described above.

Generation comes in two broad flavours. Symbolic generation works with notes rather than sound. The folk-tune generator FolkRNN, which appears in the harmony and melody chapter, is a good example: trained on thousands of transcribed Irish tunes, it writes new melodies in notation that a musician can play, edit, or discard. Audio generation skips the score and produces a finished recording directly. Research systems such as MusicLM from Google and MusicGen from Meta showed that a model can turn a written prompt like “upbeat funk with slap bass” into sound. Commercial text-to-music services such as Suno and Udio now deliver complete songs with vocals from a sentence of text.

How do such systems learn? They process huge collections of music, gradually adjusting their internal weights until they can predict how such music tends to continue. When prompted, they assemble new output that follows the statistical patterns they have absorbed, rather than replaying any single piece from the training set.

The training set is where the debate starts. Much of the music these systems learned from is copyrighted, and the artists who made it were generally not asked for consent, credited, or compensated Sturm et al., 2019. In 2024, major record labels took leading text-to-music services to court over the use of their catalogues, and that litigation is still ongoing. There is also a cultural worry: models trained mostly on commercially successful Western repertoire pull their output towards that average, and a flood of generated music could crowd out the very diversity the models learned from.

What are these systems good at? Today they excel at idiomatic pastiche: a plausible thirty-second lo-fi beat or a jingle in the style of 1980s synth-pop. They are much weaker at long-range form, at developing a musical idea over several minutes, and at the deliberate rule-breaking through which styles change. Many artists therefore treat them as tools rather than replacements, generating raw material to sample and rework or sketching arrangements, much as samplers and drum machines were absorbed into practice after controversies of their own. Whether that comparison holds is one of the questions the next section turns to.

Ethics of audio data, listening, and machine learning¶

Machine listening and AI rest on datasets and models with social consequences. As you experiment with features and classifiers, keep the following issues explicit:

• Consent and privacy — Field recordings, concerts, and classroom taps may contain identifiable voices or protected contexts. Sharing audio without clear consent can harm individuals and communities.
• Bias — Training data skew (genre, language, recording quality, Western notation-centric labels) produces models that fail silently for underrepresented musics or performers.
• Training data and generation — the generative systems described above raise unresolved questions of artist consent, credit, and compensation, which ongoing litigation is now testing.
• Surveillance — Automatic mood detection, speaker identification, or copyright enforcement can be misused; technical literacy includes questioning deployment contexts.
• Environmental cost — Large models and long GPU jobs have energy footprints; proportionality matters in research design.
• Transparency — When AI assists writing or analysis (see the introduction), disclose use where academic integrity rules require it and retain responsibility for claims you publish.

These points do not forbid computational work; they frame responsible practice alongside feature extraction and evaluation metrics.

Chapter summary¶

Machine listening treats audio as data. Preprocessing, features, machine learning, and evaluation connect signal processing to tasks such as classification, transcription, and source separation, and they are always intertwined with dataset bias, ethics, and interpretation. The same pipeline scales from single pieces to rooms and whole collections, and ambisonic recordings add direction and diffuseness to the descriptive toolbox. Generative systems reverse the pipeline, learning from large corpora to produce new notation or audio, and sharpen the debates about consent and copyright.

Questions¶

1. What stages typically make up a machine-listening pipeline from waveform to prediction?
2. Name several families of audio features and what musical or perceptual aspects they approximate.
3. In the worked example, why did the key and tempo estimates need to be read critically, and what does the case suggest about the difference between acoustic description and musical interpretation?
4. How does generative music AI differ from analytical machine listening, and what are current text-to-music systems good and bad at?
5. What ethical issues (consent, bias, surveillance, environmental cost) should accompany technical choices in MIR and ML, and whose interests are at stake when a generative model is trained on copyrighted recordings without the artists’ consent?