6. Sound, music and AI

This chapter works at the practice layer of the five layers. It begins where machine listening ends, at the turn from analysing sound to producing it. If you have spent any time computing spectrograms, extracting features, or training a classifier on audio, you already have most of the machinery this chapter needs. What changes is the direction you run it in.

Sound arrived late. Text and images reached the public in 2022, and audio of comparable quality took another year or two. By 2024 there were text-to-music services that produced finished songs from a sentence, and voice cloning that worked from half a minute of reference audio. By 2026 both sit inside ordinary podcast, film, and game workflows, which is why this chapter is about using them rather than about admiring them.

It is also the chapter where consent stops being abstract. A voice belongs to a body, it identifies the person it came from, and a convincing copy of it can be used to defraud, harass, or impersonate them. Chapter 3 sets out the general argument about training data and consent; this chapter is about what it means when the material is somebody’s speaking voice and you are the one holding the microphone.

From analysis to generation¶

An analysis pipeline takes sound and turns it into something smaller: a spectrogram, a set of features, a sequence of codes, a label. A generative pipeline does the same journey in reverse. It produces the smaller thing and then reconstructs sound from it. The representation in the middle is the same object in both cases, which is why the two halves of this field are much closer than their separate literatures suggest.

Take the three representations you are most likely to have met. A spectrogram is a picture of how energy is distributed across frequency over time; an analysis system reads it, and a generative system writes one and then inverts it back to a waveform. A neural audio codec compresses sound into a short sequence of discrete codes; a classifier can be trained on those codes, and a generator can predict them and hand them to the codec’s decoder. Features such as pitch, loudness, and spectral centroid summarise a signal for analysis, and the same curves work as control signals telling a synthesis model what to do next.

So the honest description of generative audio is not a new field beside machine listening, but machine listening run backwards through the same representation. That has a practical consequence for you: a good ear for what a representation throws away is also a good ear for what a generator will get wrong. A spectrogram discards phase, which is the alignment in time of the components that make up a sound. A model that works on spectrograms therefore has to have the phase invented for it on the way out, and that invention is exactly where the watery, smeared quality of some generated audio comes from.

An autoencoder is a pair of networks: one compresses an input into a short list of numbers, and the other rebuilds the input from that list. The short list is the latent representation of chapter 2, and training rewards a reconstruction that resembles the original. The clearest demonstration of the turn is RAVE, a variational autoencoder for audio published as open research in 2021 Caillon & Esling, 2021. It is trained on a corpus of sound and learns a compact latent representation of it, with an encoder that maps incoming audio into that space and a decoder that maps it back out. Train it on violin recordings, feed your own singing into the encoder, and the decoder reconstructs your gestures in the timbre of the corpus. That is timbre transfer: your phrasing, someone else’s instrument.

What makes RAVE matter here is not the quality of any single output but its size. The model is small enough to run faster than real time on an ordinary laptop processor, which means the transfer happens while you sing rather than after you have finished. The weights and the training code are published, so you can train one on a corpus you recorded yourself, and researchers and musicians have. Chapter 2 described the compressed spaces such models learn, and this is what one sounds like when you point it at your own voice.

Three families of audio AI¶

At least three quite different things travel under the heading of generative audio, and confusing them is the most common way to pick the wrong tool.

1. Text-to-speech and voice cloning. A synthetic voice reading text you supply, optionally in the voice of a specific person captured from a short reference recording. Quality was transformed first by neural vocoders and then by transformer-based systems trained on very large speech corpora.
2. Music generation. Systems that produce a piece of music, often with vocals and lyrics, from a paragraph of prompt. The public research line runs through work such as MusicLM Agostinelli et al., 2023; the systems most students will actually use are commercial text-to-song services built on the same ideas.
3. Sound effects and sound design. Short generated events and atmospheres for film, game, and podcast work: footsteps on gravel, a door in a large hall, rain on a tin roof, an alien room tone.

A fourth family, speech recognition, is analysis rather than generation, so it belongs to the machine listening side of the boundary. It appears in this chapter only because it sits at the front of almost every practical audio workflow, including the one in the lab.

Underneath all three families is the pattern chapter 2 set out for every generative model. A system trained on a large quantity of audio learns to predict how such audio tends to continue, and at inference time it produces something that resembles the distribution it was trained on. The differences between the families are differences of representation, of conditioning, and of what counts as a good result, not differences of principle.

Figure: A spectrogram of a synthesised signal holding two plucked notes, a noise burst, and a rising chirp, with time along the horizontal axis, frequency up the vertical one, and brightness showing the level at each point. Many audio models treat generating sound as generating an image of this kind and then converting it back.

How models represent sound¶

Computers store sound as a long sequence of numbers, typically 44 100 or 48 000 of them per second per channel. That is an awkward quantity for a generative model. Producing audio directly, one sample at a time, is what the first widely known neural audio model did in 2016, and the results were remarkable and far too slow to use.

Modern audio models almost always work on a compressed representation instead. Two are common.

• A spectrogram, the time by frequency image above. Treat the image as an image, run a diffusion or transformer model over it, then convert the result back to a waveform with a vocoder, a network trained to do exactly that inversion.
• A discrete code sequence from a neural audio codec, a small autoencoder trained to compress audio into a handful of code streams and decode them again. The generative model predicts codes, and the codec turns them into sound.

Either route makes the sequence the model has to produce smaller by something like one to two orders of magnitude compared with raw samples. That reduction, more than any single architectural idea, is why a laptop can now produce a minute of music in seconds.

The backbone is usually a transformer Vaswani et al., 2017, adapted to the very long sequences audio implies, or a diffusion model operating on spectrograms or on codec tokens Agostinelli et al., 2023. The idea of learning a representation of sound rather than hand-designing one goes back further. A 2017 model trained an autoencoder on individual instrument notes and showed that you could interpolate smoothly between two timbres in the learned space Engel et al., 2017. That is the ancestor of the latent-space instruments later in this chapter.

Each representation has its own failure signature, and learning to hear them is a genuinely useful skill. Codec-based systems tend to produce artefacts that sound like low-bitrate streaming, a slight granularity or swirl in cymbals and in sibilants, the hissing consonants of speech. Spectrogram-based systems tend to smear transients, the sharp attacks at the beginnings of notes and words, because a sharp attack is a narrow event in time that the vocoder has to reconstruct without the original phase. If you can tell those two apart by ear, you can usually tell which family of system produced a clip.

Speech, music and sound design¶

Text-to-speech and voice cloning¶

A modern text-to-speech system takes two things: a piece of text, usually converted to phonemes on the way in, and a speaker embedding, a vector describing the desired voice. The embedding can come from a catalogue of stock voices or be extracted from a reference recording, and a few tens of seconds of clean speech is often enough. From those two the system produces a waveform.

What works well in 2026:

• Convincing prosody in English and the major European languages, Norwegian included with the right model.
• Cloning a specific voice from a short reference. This works considerably better than most people expect, which is the subject of the next section.
• Style and emotion control, either through a prompt, through tags in the text, or by supplying a reference clip in the delivery you want.
• One voice across several languages, without the speaker recording anything in the new language.

What still struggles:

• Coherence over long durations. A twenty-minute reading drifts in tone and energy, and the drift is easier to hear than to fix.
• Singing and the boundary between speech and song. Dedicated music systems handle this better than speech systems do.
• Code-switching inside a sentence, especially with technical terms, proper nouns, and abbreviations.
• Low-resource languages. Sámi languages, Faroese, and a great many African and Asian languages remain badly served, for the reasons chapter 4 sets out about training data.

Norwegian is a partial exception on the recognition side, because of a deliberate public effort rather than commercial interest. The NB AI Lab at the National Library of Norway trains speech models on the library’s own archive of recorded Norwegian and publishes the weights openly. Its NB-Whisper speech recognisers handle both written standards and a range of dialects while running on a laptop Norwegian National Library, 2024Norwegian National Library, 2024. Chapter 4 makes the wider argument about why a small language needs open models rather than a hosted service.

A note on consent¶

You can clone a recognisable version of someone’s voice from a couple of minutes of publicly posted video. That is a statement of what is currently possible, not a suggestion.

Two habits are worth forming now, because they cost nothing and are expensive to retrofit. Keep the consent in writing alongside the audio, saying who agreed to what. And keep a provenance note with every generated file, recording the source, the tool, and the date. The lab asks for both.

Music generation¶

Music generation is harder than speech, for three reasons that are worth separating.

The structure is longer-range. A sentence is coherent over seconds; a song is coherent over minutes, across verses, choruses, a build, and a return, and predicting the next moment well does not make the third minute follow from the first.

The judgement is aesthetic. A wrong word is wrong in a way you can point at. A wrong note is a stylistic decision until proven otherwise, so the usual evaluation machinery has much less to bite on, and listening tests carry more of the weight than they can comfortably bear.

The training data is contested. Music is densely copyrighted, the recordings these systems learned from were generally used without the consent of the artists who made them, and in 2024 major labels took leading text-to-song services to court over exactly that Recording Industry Association of America, 2024. As of 2026 the litigation is unresolved. The legal and practical questions are laid out carefully in a 2019 survey that has aged well Sturm et al., 2019, and the technical background to the whole field is covered in a book-length treatment Briot et al., 2020. Artist-led responses exist too, including the Spawning coalition’s work on letting creators opt their material out of training sets Spawning, 2024.

Two kinds of music model are worth separating. A symbolic model writes notes, in a format such as MIDI, which something else then plays, so its output can be edited in a score editor afterwards. An audio model writes the sound itself, which arrives finished and is much harder to change. The services described here are audio models, and the symbolic route stays the better one when you intend to keep editing.

Despite all of that, as of 2026 a commercial text-to-song service reliably produces a two to three minute track from a paragraph, often with sung lyrics. It will usually let you export separated stems, meaning the individual vocal and instrument tracks, for further editing. The prompt elements that actually change the result are consistent across services:

• Genre and era, for example 1970s funk, Norwegian black metal, modern indie folk.
• Instrumentation, for example fingerpicked acoustic guitar, brushed snare, double bass.
• Tempo and feel, for example 110 beats per minute, swung eighths, intimate, late night.
• Lyrics, where supported, in their own field with section tags rather than mixed into the description.

What still fails is more interesting than what works, because it tells you where a human is still needed:

• Named-artist imitation. Asking for the style of a specific living artist raises the legal and ethical problems above, and is blocked outright by most services.
• Lyrics outside English. Norwegian output has improved and remains uneven, with stress patterns that a native speaker hears immediately.
• Form beyond pop. Multi-section classical works, fugal writing, and free improvisation are all much weaker than a three-minute verse-chorus song, which is the form these systems have seen most of.

Machine listening, the study of how models learn from audio, is a field of its own, and this chapter takes from it only what it needs. The working question it leaves you with is this: the model is strong at idiomatic pastiche and weak at form, so which of those two does the piece you are making actually need?

Sound design and Foley¶

Behind speech and music sits a quieter and more immediately useful category. Sound design models generate short events and atmospheres on request: five to fifteen seconds of rain on a tin roof, a wooden cart on cobblestones, a server room hum, a crowd in a hall two floors down. Foley is the film-industry name for sound effects performed and recorded to match what is on screen after the shoot, and it is named after the sound editor Jack Foley.

This is the unglamorous workhorse of generative audio. It is less spectacular than song generation and much less morally fraught than voice cloning, because an atmosphere is not a person and rarely a recognisable copyrighted work. It is also the part of the field that has been absorbed into professional practice fastest. A sound editor who needs a specific door in a specific room has always had to either record it or dig through a library, and now has a third option.

The craft has shifted rather than disappeared. Generated events are generic in exactly the way a stock library is generic, and making a scene sound like a place is still a matter of layering, placement, and what you leave out. Chapter 8 takes up what happens when the same material has to sit in a three-dimensional scene.

Playing with a model, not just prompting it¶

Everything above assumes the same interaction: you write something, you wait, you listen, you write something else. That is a compositional stance, and a reasonable one, but it is not what most musicians mean by playing. The difference is latency, and latency is what decides whether a model can be an instrument.

Latency. A text-to-song service takes tens of seconds to return a track, so you cannot respond to what you hear while it is happening; you can only judge the result and try again. An instrument works on a completely different timescale. Designers of digital instruments usually aim for a delay of the order of ten milliseconds between an action and the resulting sound. Beyond that the sound stops feeling caused by the gesture and starts feeling like a reply to it. Between those two timescales, seconds and milliseconds, lies the whole difference between prompting and playing.

Figure: Playing and prompting sit at opposite ends of the same scale of delay, and the distance between them is what makes one an instrument and the other a composing tool.

Real-time models. Getting into the millisecond range means giving up size. A real-time audio model is small, trained on one specific corpus rather than everything, and built to process a short buffer of incoming audio and emit a short buffer of outgoing audio without ever seeing the future. RAVE is the standard example, and the reason it turns up in so many performances is that it hits the target on a laptop processor with no accelerator Caillon & Esling, 2021. There are wrappers that run models of this kind inside the patching environments (visual programming tools for sound) and plugin hosts musicians already use. That matters more than it sounds: a model that lives in a notebook is a demonstration, and a model that lives in a plugin is a device on a stage.

Watch on YouTube

Instruments built on them. Once the model runs in real time, its latent space becomes something you can reach into. The dimensions of that space are continuous controls, and they can be mapped to a fader, a pedal, a breath sensor, or the motion of a body. That turns the model into an instrument with a playing technique that has to be learned. The interesting design question stops being what to type and becomes what to map: which dimension goes to which gesture, and what the performer can therefore learn to do reliably. That is the same question chapter 12 asks about movement and sensing, and it is why this chapter sits at the practice layer. You do not evaluate an instrument by looking at one output, but by playing it for a week and asking what you can now do that you could not do before.

Where sound AI fits in a real workflow¶

Three observations hold across the studios, newsrooms, and research groups currently using these tools.

Generated audio is an ingredient, not a product. It gets imported into a digital audio workstation, where it is layered, edited, equalised, compressed, and mixed against material that was recorded. Very little ships in the state the model produced it. Judging generative audio by the raw output is like judging photography by the unedited raw file.

Stem separation changed what is possible downstream. Stem separation is the splitting of a finished mix back into its parts, so that vocals, drums, bass, and the rest arrive as separate files. Models that split a finished mix into vocals, drums, bass, and everything else have become good enough for practical use, and the technique is set out in the Machine listening chapter of Sensing Sound and Music. That means you can take a generated track you cannot get clean stems out of and make your own. It is the quiet enabling technology behind a lot of remix and post-production work, and it belongs to the analysis side of the boundary rather than the generative one.

Speech recognition is the silent revolution. Automatic transcription went from an expensive service to a free local one in about five years, and researchers, journalists, and podcasters now use it daily without remarking on it. It is also the least ethically fraught audio tool in common use, because it produces text about a recording rather than a new recording of a person. It is the first step in this week’s lab.

pip install openai-whisper
whisper my-clip.mp3 --model small --language Norwegian

pip install audiocraft
python -c "from audiocraft.models import MusicGen; MusicGen.get_pretrained('small')"

This week’s lab: Explore, Reflect, Create¶

This lab runs the chapter’s own turn in miniature: analyse a recording, generate from it, and then argue about what the generation lost.

Explore (about 30 min)¶

Transcribe and re-voice. Work with a recording you have the right to use.

1. Record, or choose, a 30 to 60 second clip of speech. If it is somebody else’s voice, get their permission first and write down what they agreed to. Your own voice is the simplest option and makes the consent question easy.
2. Transcribe it with a speech recogniser, either locally using the listing in the Dig deeper note above or through a web service. Note how long it took and how many corrections the transcript needs.
3. Generate a new synthetic voice reading the same text, using a stock voice rather than a clone of anyone.
4. Put the two side by side and listen twice: once for what the transcription got wrong, and once for what the synthetic reading adds and removes. Pay particular attention to breaths, hesitations, and emphasis, because those are where the difference lives.
5. Record the comparison in your log, with the tools, the models, and the date.

Optional. Run both the original and the synthetic version through the live spectrogram from Sensing Sound and Music and see whether the difference you heard is visible.

Reflect (about 15 min)¶

Work in pairs, then in plenary. This is a discussion, not a writing block.

1. Play each other a 30-second generated music clip, without saying where it came from. Each of you says what gives it away and, harder, what does not. Try to name the cue: is it the mix, the vocal, the drums, the way the section changes, or something you cannot locate?
2. Take one position each on voice cloning as a free and universally available service, and argue it properly for three minutes: what changes for journalism, for political advertising, for a person whose voice is their livelihood, for your own recordings.
3. Listen again to the clip your partner played, and ask what it would do to somebody who was not making it. Music is used deliberately in therapy, in care homes, and in hospitals, where the question is not whether a listener can tell that a track is generated but whether it helps. Name one setting where you would use a generated piece, and one where you would refuse.
4. Close the round by stating aloud the brief for the piece you are about to make: what it is for, how long, and what it has to do. Say it as a job, not as a mood.

Create (about 45 min)¶

A 30-second piece. Make one finished thirty seconds of audio combining generated music and a generated ambience bed.

1. Take the brief you stated in the Reflect round, for example thirty seconds of background music for a research lab promotional video.
2. Generate music with a text-to-song service and iterate the prompt until you have something usable. Keep every prompt you tried, not just the one that worked.
3. Generate a separate ambience or sound-effect bed, and treat it as a layer rather than a decoration.
4. Mix the two in any free multitrack editor. A free browser-based multitrack editor that needs no account and no install will do, and the tools page names current options. If you have never opened one, the whole job is four moves: put the music on one track, put the ambience on a second, pull the ambience down until it sits under the music, and trim both ends. A first mix of thirty seconds takes about ten minutes.
5. Write a consent and provenance note: every tool and model with its version, the prompts, the edits you made by hand, and the time taken. For any voice or recorded sample, say where it came from and whether you have the right to publish it.
6. Export the mix and commit it to your portfolio with the note beside it.

A3 starts this week. Multimodal mini-piece is set now and due in week 8: three to five pages or slides combining text and at least one other data type, plus a reflection. The thirty-second piece is a candidate component of A3, not A3 itself. If you want the audio to end up in your submission, decide this week what the audio is for, because a sound bed with no piece attached is much harder to place in week 8 than in week 6.

At home, write this week’s entry in your practice log using the practice log template.

A critical look: can listeners tell AI music from human music?¶

The claim. Listeners can no longer distinguish generated music from music made by people, so the distinction has stopped mattering.

The evidence. The headline number comes from an industry survey published in November 2025 by a streaming service and a polling company Deezer and Ipsos, 2025. It reported that the overwhelming majority of respondents could not correctly identify fully generated tracks when they heard them. Taken at face value that is a striking result, and it was reported as one. Smaller controlled studies suggest a more qualified picture, in which trained listeners tend to do better than the general population and longer excerpts help everyone, though that literature is thin beside the headline survey.

The method. Look at how the striking number is produced. Respondents hear short excerpts, usually thirty seconds, usually pop, usually through whatever hardware they happen to have and often at streaming bitrates, and they make a forced choice between two options. Every one of those decisions makes detection harder. Thirty seconds of pop is the format generative systems are strongest at and the format in which a human production is most conventional, so the two are being compared in the narrow region where they are most alike. A forced choice also converts uncertainty into a guess, which is not the same as being fooled. And a survey run by a company that also announces how much generated music it is filtering out is not a neutral instrument, whatever the polling partner’s standards.

The limits. Short pop excerpts are the easy case for a generator, and the easy case is being reported as the general case. The weaknesses are elsewhere. Long-range form is where these systems are demonstrably weak, because coherence across four minutes is not what a model trained to predict the next moment optimises for. A thirty-second window is exactly the window in which that weakness cannot show. Deliberate rule-breaking is the other gap: a model trained to produce what is likely will not produce the choice that is unlikely and right, which is a fair description of most of what changes a style. The Machine listening chapter of Sensing Sound and Music reaches the same conclusion from the analysis side, and both limits, along with the questions of law and practice that come with them, are long-standing in the literature Briot et al., 2020Sturm et al., 2019.

There is also a question the survey format cannot ask. “Can you tell?” is not “does it hold up?”, and a track that survives a forced choice at thirty seconds may still be one nobody returns to. The useful version is narrower and answerable: for the job you have, at the length you need, with the listeners you have, does the difference matter? That has a different answer for a fifteen-second background bed than for a piece somebody is expected to sit and listen to, and you can settle it in this week’s lab.