Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

12. Embodied AI

Embodiment, musicking robots and the limits of the model

University of Oslo

This chapter works at the practice and culture layers of the five layers, and it is the only chapter in the book that joins those two. The course ends where its research home begins, with bodies: with instruments that are objects, rooms that have acoustics, and machines that make sound by moving something.

Every system in the eleven chapters before this one has been disembodied. You typed, spoke, or uploaded, and something came back as a file. Chapter 11 loosened that by putting a model in a loop with tools, and it closed by noticing that a loop with a deadline is a different kind of thing. This week takes that seriously. A body is not a peripheral bolted onto a model. It changes what a system can know, what it can be held to, and what counts as a good output.

Two questions run through the week. What does having a body do for a creative system, and what, after twelve weeks, do you think stays human? You rehearse your semester project this week, so the second question is not rhetorical.

Why a body?

Start with the thing that is easy to say and hard to hold on to: perception and action are one loop. You move your eyes to see, turn your head to place a sound, and run your fingers across a surface to feel its texture. Perception is something you do, not something that happens to you, and every percept you have is partly a consequence of a movement you made in order to have it.

A model trained on a corpus has none of that. Its inputs are the recorded outputs of somebody else’s actions, arriving in an order that no behaviour of its own produced. It can learn the statistics of how the world looks after people have acted in it. What it cannot do is find out what happens if it tries something, because it has nothing to try anything with. Chapter 2 described training as a loop over data; the loop in this chapter closes through a room.

The philosophical version of the argument is older than the technology, and it is known as the extended mind thesis. Clark and Chalmers proposed in 1998 that a cognitive process does not have to stop at the skin Clark & Chalmers, 1998. If an external resource is reliably available, readily used, and trusted in the way you trust your own memory, then on their account it is part of the cognitive system rather than an input to it. Their example was a notebook; the same argument has been applied since to instruments, sketchbooks, and, more recently and more contentiously, to generative tools. The claim is disputed, and you should hold it loosely, but it is a useful lever: it makes “where does the thinking happen” an empirical question rather than an obvious one.

The broader position is 4E cognition, the view that cognition is embodied, embedded, enactive, and extended Newen et al., 2018. Embodied means that the shape and the physiology of the body take part in thinking rather than merely carrying it. Embedded means that cognition is fitted to a physical and social setting and does not transfer cleanly out of it. Enactive means that meaning is brought forth through activity rather than decoded from a stimulus. Extended is the Clark and Chalmers claim above. The framework has real critics, who point out that the four terms overlap and that the evidence for the fourth is thinner than for the first, so treat it as a productive lens rather than a settled result.

What does a body actually buy a creative system? Three things, and it is worth separating them, because they are often argued about as one. It buys self-generated data, since a machine that acts gets to see the consequences of its own actions rather than only of ours. It buys constraints, because physics rules out most of what the software could otherwise ask for, and constraints are what make a search space searchable. And it buys legibility, because a person in the room can see the machine gathering itself to strike, and can therefore play with it. None of the three requires the machine to understand anything, which is why the rest of this chapter can be concrete without settling that argument.

Movement and mapping

Movement, motion, action, gesture

Four words get used interchangeably in ordinary speech and cause a great deal of confusion in this field, so this course follows the terminology of Sound Actions Jensenius, 2022.

  • Movement is the experienced phenomenon, the body travelling through space as you feel and perceive it.
  • Motion is the measurable version, the physical displacement of an object in time, the thing a sensor records.
  • Action is a chunk of motion with a fuzzy beginning and end, perceived as one coherent unit, such as a drum stroke.
  • Gesture is the meaning-bearing part, the component that communicates something to somebody.

The word gesture comes from the study of communication, where it names the movements that accompany speech and carry part of its meaning. A speaker’s hands, timing, and posture do work that the words do not, and the same gap between what is measured and what is meant opens there. A system that logs a hand rising has motion, and what the hand meant is a further claim.

The distinction that matters for this chapter is between what is measured and what is inferred. A robot’s sensors give it motion, which is continuous and has no natural boundaries in it. Everything the robot needs in order to respond musically lives at the level of action and gesture, which are not in the signal. Segmenting a stream of motion into actions is an interpretation, and attaching meaning to those actions is a further one. When a system is described as doing “gesture recognition”, it is doing motion capture, then segmentation, then classification, and the word gesture is a claim about the last step rather than a description of the first.

Action-sound couplings and mappings

On an acoustic instrument, what you do and what you hear are linked by physics. Hit the drum harder and it is louder and brighter; the relationship holds every time, varies continuously, and was not designed by anybody. Sound Actions calls these action-sound couplings, and their consistency is why an instrument can be learned by playing it rather than by reading its manual Jensenius, 2022.

Electronic and digital instruments have no such law. The link between an action and the resulting sound is invented by whoever built the instrument, and Sound Actions calls the invented link an action-sound mapping. Any action can be attached to any sound, which sounds like freedom and is mostly a problem, because a mapping with no continuity cannot be learned. If a small change in what you do produces an unrelated change in what you hear, there is nothing for practice to get better at.

Source
import matplotlib.pyplot as plt
from matplotlib.patches import FancyBboxPatch

navy, purple, rose = "#1f2545", "#5a2a7a", "#e8556d"

fig, ax = plt.subplots(figsize=(10, 4.4))


def box(x, y, w, text, outcome=False):
    ax.add_patch(FancyBboxPatch((x, y), w, 0.9,
                                boxstyle="round,pad=0.02,rounding_size=0.12",
                                facecolor="#fff0f3" if outcome else "#f4f1fa",
                                edgecolor=rose if outcome else purple, linewidth=1.8))
    ax.text(x + w / 2, y + 0.45, text, ha="center", va="center", fontsize=11.5, color=navy)


ax.text(0.4, 4.6, "Action-sound coupling: fixed by physics", fontsize=12,
        fontweight="bold", color=purple)
box(0.4, 3.3, 2.5, "Body")
box(7.0, 3.3, 3.0, "Sound source", outcome=True)
ax.annotate("", xy=(7.0, 3.75), xytext=(2.9, 3.75),
            arrowprops=dict(arrowstyle="-|>", color=navy, lw=2))
ax.text(4.95, 4.0, "coupling", ha="center", fontsize=11, color=navy)
ax.text(4.95, 3.4, "lawful, continuous, learnable", ha="center", fontsize=10,
        color="#444444")

ax.plot([0.4, 11.6], [2.85, 2.85], color="#cfc7dd", lw=1)

ax.text(0.4, 2.4, "Action-sound mapping: invented by a designer", fontsize=12,
        fontweight="bold", color=purple)
box(0.4, 1.0, 2.2, "Body")
box(3.2, 1.0, 2.0, "Sensor")
box(5.8, 1.0, 2.4, "Computer")
box(8.8, 1.0, 2.2, "Speaker", outcome=True)
for x1, x2 in [(2.6, 3.2), (8.2, 8.8)]:
    ax.annotate("", xy=(x2, 1.45), xytext=(x1, 1.45),
                arrowprops=dict(arrowstyle="-|>", color=navy, lw=2))
ax.annotate("", xy=(5.8, 1.45), xytext=(5.2, 1.45),
            arrowprops=dict(arrowstyle="-|>", color=rose, lw=2, linestyle="--"))
ax.text(5.5, 2.0, "mapping", ha="center", fontsize=11, color=rose)
ax.text(5.9, 0.45, "chosen, and only learnable when it is continuous",
        ha="center", fontsize=10, color="#444444")

ax.set_xlim(0, 12)
ax.set_ylim(0.2, 5.0)
ax.axis("off")
plt.tight_layout()
plt.show()
<Figure size 1000x440 with 1 Axes>

Figure: A coupling runs straight from the body to the sound source and is fixed by physics, while a mapping passes through a sensor and a computer and is chosen by whoever built the instrument.

Read the whole course through that pair and the generative systems of chapters 4 to 7 look like mappings with very poor continuity. A prompt is an action, and the output is a sound or an image, but two neighbouring prompts can give unrelated results, and the same prompt twice can too. That is the deep reason prompting feels less like playing than like negotiating, and it is one of the things a body changes, because a sensor stream is continuous whether the model likes it or not.

The design question then becomes what to do with a mapping that is neither direct nor lawful. Riaz, Erdem, and Jensenius (2026) take this up for embodied AI systems in everyday environments, where the mappings of interest are inverse and indirect rather than one to one Riaz et al., 2026. The everyday framing is the sharp part. A laboratory can ask a person to hold still, wear markers, and stay inside a capture volume. A kitchen cannot, so a system that lives there gets noisy, partial, indirect evidence about what somebody is doing, and has to work backwards from it. That is a harder mapping problem than any instrument presents, and it is where a great deal of embodied AI is heading.

Machines with bodies

Musicking robots

Musicking is Christopher Small’s term for music as something people do together rather than as a work to be contemplated Small, 1998, and a machine that plays with somebody is being asked to join in. Robotic musicianship is an established field with its own survey literature, and Bretan and Weinberg’s 2016 overview is still the sensible way in Bretan & Weinberg, 2016. The field has two hard halves that are usually confused. One is machine musicianship: listening, deciding, and generating something appropriate, which is software. The other is mechanical sound production: getting a motor to move a mass fast enough, quietly enough, and repeatably enough to make the sound the software decided on. The second half is why robotic musicianship is not simply generative music with a chassis.

Craig Vear’s Embodied Musicking Robots framework is the most useful conceptual account for this course Vear, 2021. Vear is interested in robots that co-create with an improvising human in real time, which rules out generate-then-play designs immediately. The architecture is subsumption-style, with layered behaviours where fast low-level responses run underneath slower deliberative ones, so that the machine always has something to do while it works out what to do. The idea worth carrying away is what Vear calls the belief system: the collection of behaviours that lets a human musician perceive the robot as a partner. Note where the claim sits. It is not a claim about the robot’s inner life. It is a claim about what a person in the room can reasonably believe while playing, and that is a design target you can actually aim at.

Source
import matplotlib.pyplot as plt
from matplotlib.patches import FancyBboxPatch

navy, purple, rose = "#1f2545", "#5a2a7a", "#e8556d"

fig, ax = plt.subplots(figsize=(10, 4.4))
ax.text(6.0, 5.3, "The robot", ha="center", fontsize=12,
        fontweight="bold", color=purple)

stages = [(0.8, "Sense", "microphones, cameras"), (4.7, "Decide", "listen, segment, choose"),
          (8.6, "Act", "motors and solenoids")]
for x, name, gloss in stages:
    ax.add_patch(FancyBboxPatch((x, 3.9), 2.6, 1.0,
                                boxstyle="round,pad=0.02,rounding_size=0.15",
                                facecolor="#f4f1fa", edgecolor=purple, linewidth=1.8))
    ax.text(x + 1.3, 4.55, name, ha="center", va="center", fontsize=12,
            fontweight="bold", color=navy)
    ax.text(x + 1.3, 4.18, gloss, ha="center", va="center", fontsize=10, color="#444444")
for x1, x2 in [(3.4, 4.7), (7.3, 8.6)]:
    ax.annotate("", xy=(x2, 4.4), xytext=(x1, 4.4),
                arrowprops=dict(arrowstyle="-|>", color=navy, lw=2))

ax.add_patch(FancyBboxPatch((4.4, 0.9), 3.2, 1.1,
                            boxstyle="round,pad=0.02,rounding_size=0.15",
                            facecolor="#fff0f3", edgecolor=rose, linewidth=1.8))
ax.text(6.0, 1.62, "Human player", ha="center", va="center", fontsize=12,
        fontweight="bold", color=navy)
ax.text(6.0, 1.22, "in the same room", ha="center", va="center", fontsize=10,
        color="#444444")

ax.plot([9.9, 9.9, 7.9], [3.9, 1.45, 1.45], color=rose, lw=2)
ax.annotate("", xy=(7.6, 1.45), xytext=(8.0, 1.45),
            arrowprops=dict(arrowstyle="-|>", color=rose, lw=2))
ax.text(10.1, 2.7, "sound in the air", fontsize=10, color=rose)

ax.plot([4.4, 2.1, 2.1], [1.45, 1.45, 3.6], color=navy, lw=2)
ax.annotate("", xy=(2.1, 3.9), xytext=(2.1, 3.6),
            arrowprops=dict(arrowstyle="-|>", color=navy, lw=2))
ax.text(1.9, 2.7, "playing, moving", ha="right", fontsize=10, color=navy)

ax.text(6.0, 0.4, "The loop closes through the room, so latency and the physics of the "
        "body set its pace.", ha="center", fontsize=10, color=navy)

ax.set_xlim(0, 12)
ax.set_ylim(0.1, 5.7)
ax.axis("off")
plt.tight_layout()
plt.show()
<Figure size 1000x440 with 1 Axes>

Figure: A musicking robot senses, decides, and acts, and the loop closes through the room and the person playing in it.

RITMO has built three families of musicking machines, and they divide the problem differently.

ZRob is a robotic drummer developed at UiO between 2020 and 2024, with Mojtaba Karbasi as the PhD fellow on it, supervised by Alexander Refsum Jensenius, Jim Tørresen, and Rolf Inge Godøy RITMO, 2024. Its arm uses quasi-direct-drive servo motors, 3D-printed parts, and passive springs, and the springs are the interesting decision. Rather than fighting the drum membrane with a stiff, fully controlled actuator, the design exploits the natural dynamics of the membrane, so some of the drumming behaviour comes out of the mechanics rather than out of the controller. On top of that, the project studies how a machine can learn to drum through interaction with its environment, using analysis by synthesis, reinforcement learning, curiosity-based learning, and intrinsic motivation Karbasi et al., 2023. Read those two halves together and ZRob is an argument about where musicianship lives: partly in the learning, and partly in the springs.

Two robotic arms holding drumsticks over a snare drum and a wooden practice pad in a dark laboratory, with springs, motors, and coloured wires visible.

Two ZRob arms poised over a drum kit, with the springs that let the mechanics do part of the drumming. Photo: ZRob project, RITMO, University of Oslo.

Dr. Squiggles is a swarm of small autonomous musical robots built between 2020 and 2022, designed by Michael Krzyżaniak with Kyrre Glette, Çağrı Erdem, and Jensenius RITMO, 2022Krzyżaniak, 2021. Chapter 11 treated one of these as a loop. Here they are bodies. Each unit makes sound by physically tapping whatever it is standing on, with limbs that have mass and travel time, so the instrument is partly the robot and partly the table. Because the tapping is visible, the units are legible in the sense of Why a body? above. A person in the room can see a limb rise before they hear it land, which is most of what makes playing along with them possible. Put several in a room and you have several visible, audible agents rather than a mix coming out of a speaker, and people orient to them the way they orient to other players.

Two small robots shaped like octopuses, one with a red knitted body and one with a blue one, each with a glowing dot-matrix face and solenoids mounted on black tentacles.

Two Dr. Squiggles units, whose solenoid limbs make sound by tapping whatever surface the robots stand on. Photo: Dr. Squiggles project, RITMO, University of Oslo.

A swarm of Dr. Squiggles robots improvises with a guitarist, listening to the guitar and answering by tapping.

Watch on YouTube

The self-playing guitars run from 2016 to 2027 under Jensenius, and chapter 6 introduced them as instruments: acoustic guitars fitted with microcomputers, sensors, and actuators, so that generated sound radiates from a wooden body RITMO, 2026. What matters here is the input side rather than the output side. The guitars sense the motion of people in the space around them, which makes the design problem an action-sound mapping problem of exactly the kind above. What should a person walking past do to the sound? Nothing at all is boring; a direct one-to-one link makes the instrument a novelty; and anything in between is a judgement about how much agency a passer-by should discover they have. The project has also taken the guitars out of the laboratory and into other spaces, which is where the everyday-environment problem from Riaz et al., 2026 stops being theoretical.

Visitors standing among acoustic guitars hung upright in a dimly lit room, some looking towards the instruments and some away.

Visitors move among the self-playing guitars, and their motion is part of what the instruments listen to. Photo: Self-playing guitars project, RITMO, University of Oslo.

The self-playing guitars set up outside the laboratory, in Tampere, where the room and the people in it become part of the instrument.

Watch on YouTube

Behind all three sits a design choice this chapter has not yet named. A robot’s behaviour can be written by hand, as rules and thresholds, or it can be learned from interaction, which is what ZRob’s drumming does. A third option is to evolve it: a population of controllers, and sometimes of body shapes as well, is scored on a task, and the best are varied and kept for the next round. That third option is the evolutionary computation of chapter 2 applied to machines that move, and evolutionary robotics is a research strand at UiO’s ROBIN group in the Department of Informatics, which co-built Dr. Squiggles. Eiben and Smith give the textbook treatment Eiben & Smith, 2015. It is worth knowing about mainly for what it needs. Evolution asks only that you can score a behaviour, not that you can differentiate it, which suits a machine in a room. There, what you can measure is often whether the task got done rather than how the controller ought to change.

Sensors and actuators as interface

A sensor-to-generator mapping is a prompt made of motion, and it fails in the opposite direction from a text prompt. A text prompt is deliberate, discrete, and finished: you decide, you type, you stop. A motion prompt is continuous, partly involuntary, and never finished, because a body in a room is always producing signal. Nothing tells the system when you meant something, which returns you to the segmentation problem from Movement, motion, action, gesture and makes it the central engineering task rather than a preprocessing step.

Latency then sets the budget for everything else. The tolerances discussed in the instrument literature are in the low tens of milliseconds, and the acceptable figure depends on the instrument and the task rather than being one number Jensenius, 2022. What follows is architectural. A model that takes half a second to respond is not a slow instrument, it is a different instrument, one you play by anticipating rather than by reacting. This is the constraint that rules out most large models on the critical path and pushes the interesting work towards small models, cached responses, and behaviours that start before the decision is finished.

The output side has physics too, which is easy to forget after eleven chapters of files. An actuator has mass and inertia. A solenoid cannot strike again before it has returned, a motor has a maximum acceleration, and a mechanism that is fast is usually also loud in ways nobody asked for. Whatever the generative model proposes, the body decides what is playable, and that limit is not a bug to be engineered away, but the same limit that gives every acoustic instrument its characteristic patterns.

Finally, a room full of sensors makes the audience an input. Cameras, microphones, and floor sensors turn everyone present into part of the instrument, whether or not they came to be part of it. That is a genuinely attractive artistic move and it is also a consent problem, of the kind chapter 3 gave you the four questions for. If your project senses people, say at the door what is being measured, whether anything is recorded, and what happens to it afterwards.

Robots in performance, care and everyday life

Musical robots are a small field, but they are a good rehearsal for a larger one. Embodied AI is beginning to arrive in warehouses, homes, classrooms, and hospitals. It arrives with the same three problems this chapter has been describing: noisy indirect sensing, a mapping from that sensing to behaviour, and a body whose physics limits what the behaviour can be. The difference is the stakes. A drum robot that misreads the beat produces a bad bar, and a machine in a home that misreads a person produces something worse.

Care settings are where this becomes sharpest, and they are the clearest example of the legibility argument cutting both ways. A physical thing that moves in a room is attributed intention far more readily than a chatbot on a screen. That is exactly what makes it useful for engagement with older adults or with children, and exactly what makes it easy to over-trust. Design that makes a machine feel like a partner is the same design that makes a person overestimate what it understands, and in a care setting the person doing the overestimating may not be in a position to check. The MishMash Centre for AI and Creativity, which chapters 1 and 3 introduced, includes work on creativity and well-being among its strands, and that is the sort of place where the question gets studied rather than asserted MishMash Centre for AI and Creativity, 2026. The same question runs in the other direction in the creative arts therapies, where a generated image or piece of music enters a therapeutic relationship rather than a concert hall. Consent, equity, and the risk of devaluing human empathy are the standing concerns there, and they remain open questions rather than settled ones.

Broadcasting meets the same problem from the other side. A presenter is a body in a room, and an audience reads a face, a breath, and a pause as evidence that somebody is there. A synthetic presenter keeps the words and removes that evidence, which is why the disclosure obligations of chapter 3 apply to it and not only to a generated clip.

Two habits transfer directly from this course. Ask what the system measures and what it infers, because the gap between the two is where the errors live. And ask who is in the room and whether they agreed to be, because an embodied system does not offer the option of closing the tab.

What stays human

Whatever the future, some things stay human longer than others. A non-exhaustive list of things that current and near-future AI cannot do well:

  • Sit in a room with another person and read their face for ten minutes. Therapists, teachers, social workers, nurses, mentors.
  • Live performance that depends on the audience being in the same physical space.
  • Long, situated, embodied research — fieldwork, ethnography, clinical care.
  • Care for the very young, the very old, the very sick.
  • Take moral responsibility for a piece of work or a decision.
  • Be liable in a court.

These are not “AI-proof” categories in the sense that AI cannot affect them, because it can and will. They are categories where the centre of gravity stays human for at least the medium term.

Read the list again after this chapter and it stops looking like a list of six unrelated things. The first four are all about bodies that are present, in a shared room, over time, in situations that nobody scripted. The last two are about accountability, which needs something that can be held responsible, and a body in a room is the ordinary way of knowing where to point. That is why the two halves of this chapter belong together. The reason a body is interesting for a creative system is the same reason the presence of a person is hard to substitute. The machines described above are useful precisely because they make the difference visible rather than hiding it behind an interface.

None of this makes an embodied system harmless or a disembodied one trivial. It does mean that the honest question about any tool in this book is not whether it could in principle do the job. The question is what is lost when the thing in the room is a mechanism rather than a person, and whether that loss is one your work can afford.

Reading the next decade

A few habits that help, regardless of what happens:

  • Read primary sources. A model card, a research paper, an EU regulation. These are easier to read than the takes about them, and they are more reliable.
  • Watch benchmarks fall. Stanford’s AI Index Maslej et al., 2024 and similar trackers follow capabilities over time. Look at the trend rather than the snapshot, and read how a benchmark was built before you believe what it says.
  • Build small things. A weekend project tells you more than a year of essays.
  • Talk to people in other disciplines. Lawyers, doctors, librarians, designers, teachers. They each see a different face of the same technology.
  • Stay sceptical of both extremes. Both “AI changes nothing” and “AI changes everything” are almost always wrong.

A note on quantum computing

One horizon question comes up often enough to deserve an answer, though nothing else in this course touches it.

A quantum computer is not a faster classical computer. It holds information in qubits, which can sit in superpositions of the two classical states, and it computes by arranging interference between the many paths a calculation can take, so that wrong answers cancel and right ones reinforce. Measuring collapses the state to ordinary bits and gives you one reading, so the whole art is designing an interference pattern that makes the answer you want likely. A few problems suit that arrangement and most do not.

Where the hardware stood in 2026 is easy to state. The devices are small and noisy, error correction would consume most of the qubits they have, so they run without it, and John Preskill’s 2018 name for this stage, the noisy intermediate-scale quantum era, still describes it Preskill, 2018. As of 2026 there is no demonstrated advantage from quantum hardware for training or running a generative model of the kind this book is about.

Research continues under the name quantum machine learning, and it is worth knowing what it proposes. Quantum kernels measure similarity between data points in a space that is awkward to reach classically. Variational circuits act as trainable models with a classical optimiser in the loop, and quantum-assisted sampling tries to draw from distributions that classical samplers find hard Biamonte et al., 2017. Schuld and Killoran argue that chasing a proof of “quantum advantage” may be the wrong goal altogether, and that the field is better pursued as a source of new kinds of model than as a race against classical computers Schuld & Killoran, 2022. That is a live disagreement, not a settled result.

There is also a small body of quantum computer music and art, collected in Miranda’s 2022 volume, which mostly uses quantum devices as sources of structure and of a particular flavour of randomness rather than as faster generators Miranda, 2022. It is genuinely interesting work, and it is not a performance claim.

So when you meet a headline about quantum AI, read it with the claim, evidence, method, and limits habit. What exactly is claimed, and on which layer? Was anything measured against a classical baseline? Did the result come from real hardware or from a simulation, and at how many qubits? And what limits do the authors state themselves? Those answers are usually in the paper and usually missing from the announcement.

Closing

When this course was designed in 2026, the field was moving so fast that two of the tools on the tools page had merged and one had been bought before the syllabus was approved. By the time you read this in your future career, every tool name will have changed.

What will not have changed is the structure of the questions:

  • How do these systems work?
  • How do you use them well?
  • Who benefits, who is harmed, and what does your practice owe them?

One more question will not change either, and it is the one this chapter has been circling, because it is not about the technology. Somebody has to be in the room, decide what is worth making, and answer for it afterwards. Twelve weeks of tools do not settle that, and they were never going to.

You leave this course with a small toolkit, a personal log of decisions, and one finished project. Take all three with you.

This week’s lab: Explore, Reflect, Create

The last session of the semester. Half of it is hands on a machine that makes sound by moving, and half of it is your own project, rehearsed in front of people who will tell you what is not landing yet.

The semester project is the exam, and you show it at the Synthetic Gallery in the exam period, so today’s rehearsal is the last time you get feedback before it counts. Bring the project in whatever state it is in. A rehearsal of an unfinished thing is more useful than a description of a finished one.

Warm-up (about 10 min, optional). Stand up. Tap a steady pulse on the table with one hand, then add an off-beat with the other, then have your neighbour tap along and try to pull them off the beat. Notice how much of the coordination is visual rather than auditory. That is the whole of Why a body? in three minutes, and it is what the rest of the session is trying to build a machine for.

Explore (about 30 min)

Two mappings, one of them yours.

  1. Open the rhythm-bot and tap a rhythm to it. Let it answer, then answer it back, and keep going for at least two minutes without stopping to think about it.
  2. Change one thing about how strongly it follows you, and play again. Write one sentence on what the change did to your playing, not to its output. This is the point of the exercise.
  3. Now build a mapping of your own. Use the sensor-to-sound mapper with a phone or a laptop, or borrow a sensor kit from the front of the room, and connect one continuous signal to one parameter of a sound. If the mapper does not work with your device, use any instrument app on a phone that responds continuously to tilt or to the position of a finger instead. One continuous input driving one parameter of a sound is all the exercise needs.
  4. Deliberately make it bad first: map a signal to a parameter with no continuity, so that small movements cause unrelated jumps. Play it for a minute and note how quickly you stop trying.
  5. Then make it learnable: one continuous signal, one parameter, monotonic, with a range you can reach. Play that for a minute too.
  6. Write three lines comparing the two, using the words motion, action, coupling, and mapping correctly. If you cannot, reread Action-sound couplings and mappings now rather than later.

Reflect (about 15 min)

Work in pairs. This is a conversation, not a writing block.

  1. Take each project in turn. List the parts that are embodied, meaning they depend on somebody being present, moving, performing, or in a particular room, and the parts that are not.
  2. For every embodied part, say what would be lost if it were delivered as a file instead. If nothing would be lost, it was not embodied, it was just live.
  3. For every disembodied part, ask whether it should be. Adding a body is not automatically better, and a good answer here is often no.
  4. Take one thing you say with your hands, your face, or your timing rather than with words. Say what a system would have to measure in order to notice it, and then say what it would still be missing.
  5. Look at your week 1 definition of Creative AI and your partner’s. Say out loud which week moved yours, and what moved it.
  6. Close by each stating whether your project is a performance or an installation, and the one thing you are going to rehearse in the next 45 minutes. Say it aloud, because saying it stops you rehearsing the part you already know works.

Create (about 45 min)

A timed rehearsal of your semester project, with real feedback.

Work in feedback groups of three or four, which need not be your project groups. Which of the two routes you take depends on how your project meets its audience at the gallery, which you named in your proposal.

If your project is a performance:

  1. Set a timer for five minutes and perform. Five minutes is the real length at the Synthetic Gallery, and it is shorter than you think. Do not skip the timer.
  2. Cover the two questions you have been answering all semester: where the AI surprised you, and where you exerted your own creative will. The audience at the gallery will ask them first.
  3. Note every technical thing that went wrong: a cable, a level, a tool that needed the network. Each one is a line on your list for the day.

If your project is an installation:

  1. Set it up in ten minutes, in a corner of the room, as close to the real thing as you can manage. Time the set-up, because that number is what you tell the people organising the room.
  2. Walk your group through it as if they had just arrived, then stand back and let them use it without you. Watch where they hesitate.
  3. Cover the same two questions, in the two-minute version you will give the panel when they reach you.

Then, both routes together:

  1. Two minutes of feedback per project, from each listener, in this form: one thing that landed, one thing that did not, and one question you would ask. Write it down for the maker rather than saying it and moving on.
  2. Commit the rehearsal notes to your portfolio before you leave the room.

At home, draft your gallery page using the gallery page template, including the format line, the technical needs, and the consent boxes. Fifteen minutes is enough for a first version, and having a draft this week is what stops the page being written at midnight. Then write the final entry in your practice log, using the practice log template. Put your week 1 definition of Creative AI at the top of the entry and today’s definition directly underneath it, unedited, then write three hundred words on the distance between them. Name the week that did the most work, and commit both files to your portfolio.

Then finish the project. The gallery is in the exam period, and the fixes the rehearsal exposed are best made in the days straight after it.

A critical look: can a robot musician be creative?

The claim. A robot that improvises with a human musician is being creative, not simulating creativity. The argument is that it produces material nobody wrote, in response to a situation nobody predicted, under a real-time constraint that rules out retrieval. The people playing with it, on this account, experience it as a contribution rather than as a playback. If that is not creativity, the argument runs, then the word is doing no work.

The evidence. The oldest counter-argument is Lady Lovelace’s. Writing about the Analytical Engine in 1843, Ada Lovelace put it this way: “The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform.” Bringsjord, Bello, and Ferrucci turned that objection into something testable in 2001, with what they call the better Lovelace test Bringsjord et al., 2001. A system passes only if it produces an output that its own architects cannot account for, given complete knowledge of the architecture, the program, and the inputs. That is a demanding bar and most generative systems do not clear it, since their surprises are explained perfectly well by sampling from a learned distribution. On the other side, work in the musicking-robots tradition reports something the Lovelace framing does not capture Vear, 2021. Human players and audiences treat these machines as partners when their behaviour is legible in the moment, and stop doing so when it is not. The variable that moves is whether you can tell what the machine is about to do, not whether its designer could explain it afterwards.

The method. Notice that those two bodies of evidence are not measuring the same thing, and that neither is a benchmark. The Lovelace test is an argument about explanation, applied by reasoning about an architecture. The musicking-robot evidence comes from performance studies: concerts and improvisation sessions, followed by interviews with the players and the audience, with the researchers often analysing a system they also built. Each method has an obvious weakness. The first can be satisfied by an unexplainable system that is merely badly documented. The second has small numbers, no control condition, and an experimenter with a stake in the answer. Ask of any claim in this area which of the two it rests on, because the strong claims usually borrow the rhetoric of one and the evidence of the other.

The limits. The honest conclusion is that the question is malformed, and it is malformed in a specific and useful way. Creativity in these settings is not a property sitting inside the machine, waiting for the right test to detect it. It is a relation between players: something that happens in the room, in the timing, between a person who is trying something and a machine whose response makes the trying worthwhile. That is this course’s answer to the question chapter 1 opened with, and it is why Boden’s kinds have held up all semester while so many capability claims have not Boden, 2004. Combinational and exploratory creativity are visible in these systems and are not seriously disputed. Transformational creativity, changing the space itself, is exactly what nobody has demonstrated and what the better Lovelace test was built to catch. So the useful question is not whether the robot is creative. It is whether the two of you, together, made something neither of you would have made alone, and that question you can answer by playing.

References
  1. Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58(1), 7–19. 10.1093/analys/58.1.7
  2. Newen, A., De Bruin, L., & Gallagher, S. (Eds.). (2018). The Oxford Handbook of 4E Cognition. Oxford University Press. 10.1093/oxfordhb/9780198735410.001.0001
  3. Jensenius, A. R. (2022). Sound Actions: Conceptualizing Musical Instruments. The MIT Press. 10.7551/mitpress/14220.001.0001
  4. Riaz, M., Erdem, Ç., & Jensenius, A. R. (2026). Inverse and indirect mappings in embodied AI systems in everyday environments. Frontiers in Computer Science, 7. 10.3389/fcomp.2025.1603769
  5. Small, C. (1998). Musicking: The Meanings of Performing and Listening. Wesleyan University Press. https://www.weslpress.org/9780819522573/musicking/
  6. Bretan, M., & Weinberg, G. (2016). A survey of robotic musicianship. Communications of the ACM, 59(5), 100–109. 10.1145/2818994
  7. Vear, C. (2021). Creative AI and musicking robots. Frontiers in Robotics and AI, 8, 631752. 10.3389/frobt.2021.631752
  8. RITMO. (2024). Interactive robotic system (ZRob). https://www.uio.no/ritmo/english/projects/ZRob/
  9. Karbasi, S. M., Jensenius, A. R., Godøy, R. I., & Tørresen, J. (2023). Exploring Emerging Drumming Patterns in a Chaotic Dynamical System using ZRob. Proceedings of the International Conference on New Interfaces for Musical Expression. 10.5281/zenodo.11189178
  10. RITMO. (2022). Dr. Squiggles: an interactive musical robot. https://www.uio.no/ritmo/english/projects/dr-squiggles/
  11. Krzyżaniak, M. (2021). Musical robot swarms, timing, and equilibria. Journal of New Music Research. 10.1080/09298215.2021.1910313
  12. RITMO. (2026). Self-playing guitars. https://www.uio.no/ritmo/english/projects/self-playing-guitars/
  13. Eiben, A. E., & Smith, J. E. (2015). Introduction to Evolutionary Computing (2nd ed.). Springer. 10.1007/978-3-662-44874-8
  14. MishMash Centre for AI and Creativity. (2026). MishMash Centre for AI and Creativity. https://mishmash.no/
  15. Maslej, N., Fattorini, L., Perrault, R., Parli, V., Reuel, A., Brynjolfsson, E., Etchemendy, J., Ligett, K., Lyons, T., Manyika, J., Niebles, J. C., Shoham, Y., Wald, R., & Clark, J. (2024). The AI Index Report. Stanford Institute for Human-Centered Artificial Intelligence. https://aiindex.stanford.edu/report/