8. The body

So far, we have treated sound and music mainly as something we hear. But we also perceive music through our bodies. This is the point where the four levels of description stop being a tidy ladder: a performer’s gesture is a physical event, a perceptual cue, and an interpretive act at the same time, and the whole argument of embodied cognition is that separating them distorts what is going on. This week looks at how the body shapes music perception and cognition: the body’s part in musical experience, some anatomy and biomechanics, and methods for analysing human movement.

Embodied music cognition¶

You met embodied music cognition at the start of this course. An embodied approach stresses the role of the body in producing, perceiving, and understanding music. Instead of treating musical experience as only an auditory phenomenon—or as something purely mental or abstract—it shows how bodily sensations, movements, and actions carry musical meaning.

Three ideas recur through the chapter: that perception and action form a loop, that musical experience draws on several senses at once, and that gesture carries meaning.

The Belgian systematic musicology professor Marc Leman popularised the term embodied music cognition in the early 2000s. In his book Embodied Music Cognition and Mediation Technology Leman, 2007, he traces the different processes at work in the bodies of both performers and perceivers, and describes how musical intentionality rests on sonic and visual communication between them:

An illustration of Marc Leman’s model of embodied music cognition (illustration from Jensenius (2022)).

4E cognition¶

In recent years, the idea of 4E cognition has gained ground, building on the “embodied turn” in cognitive science and philosophy of mind. The 4E framework—standing for Embodied, Embedded, Enactive, and Extended cognition—challenges the traditional view that cognition is confined to the brain.

Researchers such as Francisco Varela, Evan Thompson, Alva Noë, and Andy Clark argue that understanding cognition means taking the body, the environment, and their interactions into account. On the 4E view, cognitive processes are:

• Embodied: Our perception and understanding of music are rooted in the body. How we move, breathe, and physically interact with instruments or our surroundings shapes how we experience and interpret sound. Tapping a foot to a beat or feeling vibrations through the body are embodied ways of sensing music.

• Embedded: Musical experience is shaped by its context. Our interactions with sound depend on the physical and social setting: the acoustics of a concert hall, the presence of other listeners, the cultural frame. These factors embed musical cognition in a wider world.

• Enactive: We take an active part in making musical meaning. Listening is not passive; we anticipate, move, and respond, co-constructing the experience. A performer’s gestures or a listener’s dance are forms of enactive engagement.

• Extended: Tools and technologies can extend our cognitive processes. Instruments, recording devices, and even smartphones become part of the system through which we sense, produce, and understand music. Motion capture systems, for instance, extend our ability to analyse and reflect on musical movement.

The 4E perspective treats music as an interactive process that involves the whole person, situated in a specific context, working with both physical and digital tools.

Movement, motion, action, and gesture¶

Before going further, we need to sort out four words that are often used interchangeably in everyday speech and even in the research literature: movement, motion, action, and gesture. Researchers in musicology, gesture studies, human–computer interaction, and human movement science all use these terms, but often with different meanings. Does a “piano gesture” refer to physical motion, the experience of an action, a sonic phrase, or something else? In this course, we follow the terminology developed in Sound Actions Jensenius, 2022:

• Movement is the experienced phenomenon: the continuous excursions of the body in time and space, as we feel and perceive them. It is the everyday, experiential term. We talk about the “movement of a pianist’s hand” when we mean how we sense the hand travelling through the air.

• Motion is the physics of movement: the continuous displacement of an object in time and space. Motion can be measured objectively, for example with a motion capture system, and described with the kinematic quantities introduced later in this chapter (position, velocity, acceleration). Motion is continuous, with no beginning or end in nature, so strictly speaking it does not make sense to talk about “a motion.”

• Action is the subjective experience of a movement sequence: a chunk of motion with a fuzzy beginning and end. Actions are often goal-directed, but not always. Think of a drummer performing a stroke: you easily perceive it as one coherent action (lift, hit, return to rest), but the motion capture data will only show a continuous position signal. Where the action begins and ends is a matter of interpretation; nothing in the data itself marks the boundaries. Neighbouring actions also blend: a pianist’s hand is already shaping the next chord while it plays the current one. This is called coarticulation, a term borrowed from phonetics, where the mouth prepares the next speech sound before the current one has ended Godøy et al., 2010.

• Gesture denotes the meaning-bearing components of actions. A gesture is not just physical motion or a goal-directed action; it is part of human communication, what the gesture researcher Adam Kendon calls “visible action as utterance.” Consider waving goodbye: you can wave with the left hand or the right, and the physical motion will differ, but the meaning, and hence the gesture, is the same.

Norwegian has only one word, bevegelse, for both movement and motion, and so do Spanish (movimiento) and many other languages. The distinction therefore has to be made on purpose: we say movement for what we experience and motion for what we measure.

The same three-way split applies to the body at rest: position is measured, posture is how a person holds the body, and pose is a posture that carries meaning.

The most important distinction, in both triads, is between what can be measured and what must be inferred. Motion, position, and force can be captured with sensors and cameras; actions, gestures, postures, and poses can only be interpreted from those measurements. A “gesture recognition” or “pose recognition” system therefore has to start by recording motion or position, segment the continuous data stream into actions or postures, and finally try to infer their meaning-bearing components.

The Musical Gestures Toolbox, introduced in the video analysis section below, keeps these as separate steps for exactly this reason: it measures motion, segments it into candidate actions, and leaves the meaning to be added by annotation, since meaning is not a property of the signal.

The body in performance¶

When we talk about music-related body motion, it helps to separate performers from perceivers. Their roles differ, but both depend on the body to engage with music. This section looks at performers: the actions they make, how they time them, and how they coordinate with each other. The next section turns to perceivers.

Sound-producing and other actions¶

Music performance is physical. Musicians use coordinated actions, chunks of motion directed at a musical goal, to produce sound, shape phrases, and communicate with others. The body is both the source of musical ideas and their interpreter, turning intention into audible and visible action. This holds for all music-making: singing, playing instruments, conducting, and dancing.

We can sort music-related body motion in performers into four main types:

• Sound-producing actions: Movements that directly generate sound on an instrument or interface. They include selection (choosing notes or sounds), excitation (initiating sound by striking, plucking, or bowing), and modification (altering qualities such as pitch or timbre). Examples: pressing piano keys, bowing a violin, turning a synthesiser knob. These actions are the performer’s half of the action–sound couplings introduced in the acoustics chapter; on electronic instruments they instead drive designed action–sound mappings (electroacoustics).

• Sound-facilitating actions: Movements that support sound production without making sound themselves, such as holding posture, shaping phrasing, and entraining the body to rhythm. Examples: a pianist’s arm movement for dynamics, a clarinettist’s breath support, tapping a foot to keep time.

• Sound-accompanying actions: Motion that reflects or mimics musical features without producing sound, such as tracing sound contours in the air, mimicking instrumental gestures, or air performance. Examples: moving a hand upward with rising pitch, playing air guitar, dancing to music.

• Communicative gestures: Gestures that convey meaning, emotion, or instructions to other performers or the audience. They may be expressive, regulatory, or linguistic. Examples: a conductor’s baton movements, a nod to cue an entrance, expressive hand gestures.

These types of action sit along a continuum of connection to musical sound, as shown below:

Figure: The four types of music-related body motion in performers, ordered from the closest to the loosest link to the sound. After Jensenius (2022).

The categories are useful for clarity, but in practice they overlap. Sound-facilitating actions are often inseparable from sound-producing ones, and performers may use communicative gestures while playing. Keeping this interplay in view matters when studying music-related body motion.

Producing rhythm: timing in the body¶

Time and rhythm treated the beat mostly as something heard. It is also something the body makes. The two exercises in that chapter that had you tapping, first at your own rate and then along with a click, were small experiments in rhythm production, and this section looks at what such experiments show.

The standard task is sensorimotor synchronisation: tapping a finger in time with a metronome while the moment of each tap is recorded Repp, 2005. Two findings recur across hundreds of studies. First, taps do not fall on the clicks but slightly before them, by 20–50 ms for most people and by less for trained musicians. This negative mean asynchrony is a sign that tapping is prediction rather than reaction. A reaction to a sound takes at least 150 ms, far longer than the gap, so the tap must have been planned to coincide with a click that had not yet sounded. Second, the taps vary. The standard deviation of the asynchronies is typically 3–5 % of the interval, so at 120 BPM a tapper is “on the beat” only within a window of some 15–25 ms. The width of that window is roughly what separates a drummer from the rest of us.

Timing is not confined to the finger. People bounce, nod, sway, and step to music at rates around 2 Hz, the same range as the spontaneous motor tempo and the walking cadence in the time-scale figure of the rhythm chapter. The vertical bounce of a body moving to a beat is the signal simulated in the demo later in this chapter. Moving helps. Listeners who tapped along with a rhythm were better at hearing whether a later note was on time than listeners who sat still Manning & Schutz, 2013. That is one reason drummers and dancers keep the whole body in play rather than just the hand that strikes.

For performers, timing and sound are produced together. When drummers were asked to play the same groove laid-back, on the beat, and pushed, their strokes moved by tens of milliseconds as instructed, but their sound changed too. Laid-back strokes were played louder and with longer decays, so the timing style was audible in the timbre and not only in the onset Câmara et al., 2020. The body that produces a rhythm also shapes it, which is why the microrhythm section of time and rhythm could not separate timing from timbre. The reverse coupling holds for listeners, as the motion capture study in the rhythm chapter showed: bodies followed beats shifted by a few tens of milliseconds that the listeners never noticed Danielsen et al., 2015. In the telespringar study that closes the rhythm chapter, the fiddler’s foot stamps and the dancers’ vertical motion carried the same unequal beats as the music Haugen, 2016. The reference grid of that metre lives in the moving body as much as in the sound.

The measurement side follows the scale of the motion. Finger and foot timing are captured by tapping apps and sensors, such as the Spontaneous tempo and Tap along apps of the rhythm chapter, and whole-body timing by the motion capture methods described later in this chapter.

Playing together: joint action¶

Joint action in music covers situations where two or more people align perception and movement towards a shared goal in time: locking to the same pulse in an ensemble, adjusting attacks to a conductor’s gesture, breathing with a choir, or coupling dance steps to musical metre. The underlying mechanism is the entrainment of coupled oscillators described in time and rhythm; here the oscillators are people. Cognitive scientists sometimes separate shared intentionality (we mean to play “this” phrase together) from mere interpersonal synchrony (our movements happen to coincide); both turn up in rehearsal and performance.

Coordination draws on several channels at once. Auditory information alone can be enough, but visual cues such as conducting, head nods, and instrument motion, and even tactile contact or spatial proximity, all support alignment. Laura Bishop and colleagues have measured this in duos and ensembles: cueing gestures such as head nods communicate both where the next beat falls and the intended tempo Bishop & Goebl, 2018, and body motion in duos alternates between movements that communicate and movements that simply interact with the partner’s Bishop et al., 2019. Pupil size, a marker of mental effort, rises in both performers and listeners at musically demanding passages of a string quartet Bishop et al., 2021. Roles matter too. Leader–follower relationships and intentional asynchrony, such as a jazz soloist laying back behind the beat or a band pushing forward, are musically meaningful rather than errors, and musical joint action needs constant error correction, the micro-delays and anticipations that keep parts distinct yet coordinated.

Empirical work links synchrony to social cohesion and affect. Group settings nevertheless introduce confounding factors that solo studies avoid: the mere presence of others (social facilitation), the awareness of being judged (evaluation apprehension), and shared excitement can all change both movement and physiology, so claims about “bonding” or “unity” from synchrony alone need careful design and interpretation. Felt togetherness and measured coordination do not always agree: in piano duos, what the players focused their attention on changed both how together they felt and how interactive their body motion was Bishop, 2025. Togetherness can even survive distance: in networked music and dance performances between two sites, performers and audiences describe a shared presence that the researchers call performative togetherness Serdar et al., 2026.

The body in perception¶

Many of the same actions show up in perceivers, people “listening” to music. We put “listen” in quotation marks because we experience music with the whole body, which is the heart of embodied music cognition. Perceivers also make sounds during a performance, whether involuntary (breathing) or voluntary (clapping, singing along).

Moving to music¶

Perceiver motion can be voluntary or involuntary, and it shapes how we experience and understand music. Examples include:

• Dancing: Whole-body rhythmic movement, structured as in ballroom or folk dance or improvised; the most direct way to embody musical structure, rhythm, and emotion, and a part of every musical culture.
• Air performance: Imitating the actions of playing or singing, as in “air guitar” or lip-syncing, which lets a listener simulate being the performer.
• Finger-tapping or foot-stamping: Often automatic responses that keep the listener synchronised with the beat; finger-tapping is the standard laboratory measure of beat perception, as the previous section showed.
• Involuntary swaying or nodding: Subtle motion without conscious intent, seen across cultures and ages, which shows how closely the auditory and motor systems are coupled.

These bodily responses are not mere by-products of listening; they are part of musical perception and cognition. Moving to music can strengthen memory, emotional response, and social connection.

Dance couples full-body movement with musical time, metre, and phrase structure in culturally specific ways. Foot patterns, spine shape, and use of space are not decorative add-ons; they are part of how rhythm and identity are expressed. Ethnochoreology and dance studies document repertoires where “the beat” lives as much in stride and bounce as in the drums. For a listening-oriented course, the main point is that dance–music relationships are learned and stylistic—compare, for instance, club genres with contrasting upper- versus lower-body emphasis—and that motion capture and video analysis now sit alongside audio when researchers study groove.

Why we move¶

Why do we move to music at all? There is no single answer, but research points to three reasons that work together.

• Coupling: Hearing a regular pulse activates motor areas of the brain, so the body is primed to synchronise with it through entrainment. Few other animals do this. The cockatoo Snowball became famous for bobbing his head to pop songs and staying with the beat when the tempo was changed Patel et al., 2009, but such cases are rare.
• Pleasure: Moving to music is rewarding. People report the strongest urge to move, and the most pleasure, when a rhythm has a medium amount of syncopation, neither too simple nor too complex Witek et al., 2014. The reward system behind this is covered in the brain.
• Bonding: Moving together in time tends to make people feel closer to each other, which is one reason why dancing is a social activity in so many cultures.

Culture shapes all three. Which movements feel natural to a piece depends on the dances and listening habits a person has grown up with.

Micromotion and standstill¶

The involuntary end of that list is easy to assert and hard to pin down, because the movements are small and people stop making them as soon as they notice they are being watched. One way around this is to invert the task: instead of asking people to move to music, ask them to stand as still as they possibly can, and measure what is left. This is the standstill paradigm.

This is the design behind the Norwegian Championship of Standstill, run annually at the University of Oslo since 2012, in which participants stand still in groups for six minutes while an infrared motion-capture system records the position of a marker on the head. The competitive framing matters, since it removes any suspicion that participants are moving because they cannot be bothered not to.

What remains is micromotion, the residual sway of a body doing its best to hold a position. In one such study Jensenius et al. (2017), 91 participants stood still for three minutes in silence and then three minutes with music. Nobody managed to be still. The average speed of head motion was around 6.5 mm/s, and it was slightly but consistently higher with music playing than in silence.

Two things are worth taking from this. The first is the scale: the effect is measured in millimetres per second, so it is nothing you would see in a video, and claims about it depend entirely on the measurement techniques described later in this chapter. The second is the direction. Music does not have to be danced to in order to move you, and the coupling between hearing rhythm and moving is persistent enough that instructions to suppress it only reduce it. The involuntary swaying in the list above is not a figure of speech. Who sways most is not random. With the same paradigm, people high in empathic concern moved more to music than others Zelechowska et al., 2020, and listening over headphones produced more head motion than the same music over loudspeakers Zelechowska et al., 2020.

Audiences do something similar without being asked. Analyses of concert video show that listeners go still at moments of musical silence, a stilling response that can be read from ordinary video recordings Upham et al., 2024.

Feeling sound through the body¶

Haptics covers active touch and forces, and it overlaps with passive tactile vibration for musicians: fingering contact on strings or keys, stick rebound, and the resonance of an instrument body against the torso. Sound also reaches the skin directly. At a concert, the low-frequency energy of bass and kick drum radiates through floors and bodies, so you feel the drop in your chest as much as you hear it.

The percussionist Evelyn Glennie, who lost most of her hearing by the age of twelve, has built a world-class career on exactly this channel. She learned to feel vibrations through her hands, feet, and body, often performing barefoot, and uses that information to play and interpret music, as she describes in her TED talk How to truly listen:

(If the video is unavailable, try the archived page.)

Vibrotactile feedback puts the same channel to deliberate use. Digital musical instruments often map sound parameters to compliant controllers so performers “feel” the synthesis change under their hands, and wearable devices can deliver rhythmic cues through the skin. Accessibility-oriented concert designs for Deaf and hard-of-hearing audiences route timing and pulse through vibrating platforms or wearables, and when masking makes hearing ambiguous, touch can hold the pulse steady. Together with hearing—and vision when available—these channels make up multimodal musical engagement.

Anatomy and biomechanics¶

To make sense of motion capture, we first need to know how the body is built and how it moves. This section gives an overview of anatomy and biomechanics as they relate to motion analysis.

If this looks like a detour into sports science, it is worth saying why it is here. Every claim about music-related movement rests on a description of what moved, relative to what. A study of a violinist’s bowing needs to say whether the arm rotates at the shoulder or the elbow, and in which plane; a study of pianists’ injuries needs joint angles and ranges; a comparison of drummers’ strokes needs to separate the motion of the stick from the motion of the wrist that drives it. Without a shared vocabulary these descriptions become “the arm went up a bit”, which no one can replicate or measure. The terms below are that vocabulary. You do not need to memorise them, but you will meet them in every empirical paper on music and movement.

Anatomical position and planes¶

Anatomy studies the structure of the body. To keep descriptions of locations and actions consistent, we use the anatomical position: standing upright, head and eyes forward, arms at the sides with palms facing forward, and feet parallel and pointing ahead.

The human body in the anatomical position, with labelled regions.

The body is divided into regions—head, neck, trunk, upper limbs, and lower limbs—each with further subdivisions. For movement analysis, it helps to distinguish areas such as the arm and forearm, or the thigh and leg.

To describe positions and movements in three dimensions, we use anatomical planes: imaginary divisions of the body that give standard reference points for anatomical terms and motion analysis. The sagittal plane divides the body into left and right sections (the median plane runs exactly down the middle), the frontal (coronal) plane separates front (anterior) from back (posterior), and the transverse plane divides the body horizontally into upper (superior) and lower (inferior) parts. These planes let us describe the direction and type of movement precisely, in both clinical and research settings.

The three main anatomical planes: sagittal (divides left and right), frontal/coronal (divides front and back), and transverse (divides upper and lower parts of the body).

To describe movement directions, the belly button (navel) is often used as a reference point for the whole body, though other landmarks may be chosen for specific segments. Movements are usually described along three primary axes: medial–lateral (side-to-side or left–right), anterior–posterior (front-to-back), and superior–inferior (up–down). These axes are perpendicular to the sagittal, frontal, and transverse planes, and they standardise descriptions of motion in research and clinical work.

Common directions of human body motion, illustrated with arrows.

The muscular system¶

The musculoskeletal system produces and controls movement. It has two main parts: the muscular system and the skeletal system. The muscular system is made up of muscles that act on the skeleton to move or position body parts, while the skeletal system—bones, joints, and cartilage—provides structure and protection. We start with the muscles.

There are three types of muscle tissue: cardiac (heart), smooth (organs), and skeletal (attached to bones). Skeletal muscles handle voluntary movement.

A skeletal muscle has a thick, red muscle belly and narrow, white tendons at each end that anchor it to bones. When the muscle contracts, it pulls on the tendons, and the attached bone moves.

When a skeletal muscle contracts, it pulls on the attached bone via tendons, resulting in movement at the joint.

Muscles can only pull, not push, so movement usually involves several muscles working together. The agonist is the main muscle driving a movement; synergists assist it by adding force or stabilising the origin bone (sometimes called fixators); and the antagonist produces the opposite action, balancing or resisting the agonist so movement stays controlled and smooth.

With over 600 skeletal muscles, only the major superficial ones are shown here:

Major muscle groups of the human body, shown from the front (left) and back (right).

The skeletal system¶

The adult skeleton has about 206 bones, which form the body’s framework. Bones act as levers for movement and give muscles their attachment points. Many bones have distinct landmarks, features that anchor muscles and can often be felt on your own body.

Major bones and bone groups of the human body, shown from the front (left) and back (right).

Joints are the connections between bones that let the skeleton move. The structure of each joint determines its possible directions and range of motion. Knowing how joints move helps us analyse how the body produces complex actions, such as those in music performance.

Joint movements are usually described as pairs of opposite actions, always referenced from the anatomical position: flexion and extension in the sagittal plane; abduction and adduction in the frontal plane; and internal (medial) rotation and external (lateral) rotation in the transverse plane.

Major movement types at the joints, including flexion/extension, abduction/adduction, and internal/external rotation. Movements are always described relative to the anatomical position.

Degrees of freedom (DoF) are the number of independent directions in which a joint can move. Each DoF stands for a specific type of movement (flexion/extension, abduction/adduction, rotation). Range of motion (RoM) describes how far a joint can move within each DoF, usually measured in degrees. Together, DoF and RoM tell us how a joint functions, what it can do, and where its limits lie, in everyday activity and in demanding tasks like music performance.

Kinematics¶

Biomechanics applies the principles of mechanics to the body. Bodies at rest or in equilibrium, such as a standing person holding a posture, are the subject of statics; bodies in motion, such as a person walking or playing an instrument, are the subject of dynamics. Dynamics in turn splits into kinematics, which describes the motion, and kinetics, which asks what caused it. Both need a frame to describe positions in.

To analyse movement, we use a reference frame, a coordinate system for describing positions and motion. A global reference frame is fixed to the environment, such as the laboratory or stage, and acts as an external standard for measuring motion. A local reference frame is attached to a body segment, such as the hand relative to the forearm, so we can describe motion in relation to other body parts. Combining global and local frames lets us describe human movement both precisely and in context.

Kinematics describes motion—how body parts move—without considering the forces behind it. It answers what moves, where, and how fast.

Kinematics is what a motion capture system actually records: a stream of positions over time, from which everything else is derived.

Figure: Distance and displacement for the same movement. The distance is the length of the path the hand actually travelled; the displacement is the straight line from where it started to where it ended, shorter and with a direction.

Kinematic analysis lets us track motion patterns in music performance, such as the trajectory of a violinist’s bow or a pianist’s hand.

Kinetics¶

Kinetics examines the forces and torques that produce or result from movement, focusing on why and how motion happens.

Kinetics asks what caused that motion. Motion capture cannot see forces directly, so kinetics is usually inferred from kinematics plus a model of the body, or measured separately with force plates and sensors.

A person remains balanced as long as the line of gravity from their CoG falls within their base of support.

Kinetics helps us analyse how musicians generate, control, and coordinate movement, and it informs both injury prevention and performance.

Motion capture¶

There are many ways to study human body motion. Many people picture “motion capture” as suits, markers, or sensors, but the term can cover any method that systematically records movement, qualitative as well as quantitative. In practice the two are often combined. Researchers may use both video and sensors, and an analysis may mix interpretive and numerical work. For clarity, this course treats qualitative and quantitative methods separately, while recognising that mixed-method approaches are common.

Quantitative methods rely on numerical representations of motion. Video can serve as a quantitative tool too, if features are extracted and measured rather than just observed. The work often involves plotting measurements and applying statistical or machine learning techniques. Two main types of motion capture stand out: camera-based and sensor-based. Each has strengths and limits, and the choice between them depends on the research context, the precision needed, and practical constraints.

Qualitative approaches¶

Qualitative motion analysis works through observation, reflection, and descriptive frameworks rather than numerical measurement.

• Introspection: Reflecting on one’s own movement experience. Musicians and researchers may assess their own performance, notice sensations of effort or discomfort, or describe how a movement feels during music-making.
• Observation: Systematically watching others move—live or on video—and annotating their motion: posture, gesture, timing, expressiveness. Some may not count observation as “proper” motion capture, but it is a structured, repeatable way to document movement without specialised technology.

Observation-based methods are widely used in clinical, educational, sports, and artistic settings. Video recordings allow repeated viewing, slow-motion analysis, and collaborative review, which makes subtle details and patterns easier to spot.

Music researchers have drawn on the qualitative methods developed by the dancer and choreographer Rudolf Laban (1879–1958), who created two influential systems in the early to mid-20th century:

Dig deeper: Laban’s two systems

• Labanotation: A symbolic notation system for recording and analysing human movement, especially in dance. It uses standardised symbols for body parts, directions, levels, and timing, allowing detailed documentation of movement sequences.

Three sequences in Labanotation, read from the bottom up: a) walking forwards; b) walking forwards on the toes, then backwards; c) the right side of the body moving forwards and then to the right, the left side forwards and then to the left. From Jensenius (2009).

• Laban Movement Analysis (LMA): A framework for describing the qualitative side of movement. LMA covers four components: body (what moves), effort (how it moves), shape (the form the body takes), and space (where it moves). The “effort” component is especially relevant to music, describing motion in terms of space (direct/indirect), time (sudden/sustained), weight (strong/light), and flow (bound/free); a punch, for example, is direct, sudden, and strong.

Qualitative approaches capture the expressive, communicative, and contextual sides of movement. They often complement quantitative methods by reaching aspects that are hard to measure numerically, such as emotion, intention, and style.

Marker-based motion capture¶

The reference standard in movement research is optical, marker-based motion capture. Small reflective markers are attached to the body, and a ring of cameras around the room, typically 8–20 of them, films the scene in infrared light at 100–500 frames per second. Each camera sees the markers as bright dots in its own two-dimensional image; because the positions of the cameras are known from a calibration with a wand of known length, the dots can be combined into three-dimensional positions with a precision well under a millimetre. The result is exactly the kind of data that the kinematic vocabulary above describes: the position of each marker over time, in the global frame of the room, from which velocity, acceleration, joint angles, and distances between performers can be derived.

The cost of this precision is a controlled setting. The markers must be attached and later labelled, the cameras calibrated, and the room kept free of stray reflections, and a marker that a cello or the performer’s own arm hides from all but one camera drops out of the data. The systems used in music research at the University of Oslo, from the companies Qualisys and OptiTrack, record whole ensembles and dancers in this way, and the standstill studies earlier in this chapter relied on the same precision to measure sway of a few millimetres per second.

An example of an infrared, marker-based motion capture system, allowing for precise measurements of the body.

Marker-based motion capture is available at RITMO and the Department of Musicology. The video below, from the University of Oslo’s online course Music Moves, shows one of the labs:

(If the video is unavailable, try the archived page.)

Sensor-based motion capture¶

Where the cameras stay in the room, sensors go with the body. The workhorse is the inertial measurement unit (IMU), a chip the size of a fingernail that sits in every phone and smartwatch and in most wearable research devices. It combines three sensors: an accelerometer, which measures acceleration, including the constant pull of gravity; a gyroscope, which measures how fast the unit is turning; and often a magnetometer, which gives its heading relative to the Earth’s magnetic field. Combining the three, a process called sensor fusion, gives the orientation of the unit at every moment. Combined with a model of the body, a set of IMUs on the limbs gives joint angles, which is what inertial motion-capture suits do.

The strengths are portability and independence from the surroundings, which suits field studies, concerts, and situations where cameras are impractical. Sensors can also be combined with other signals on one clock: chest-worn research devices such as the Equivital carry an IMU together with electrocardiogram, respiration, and skin-temperature sensors, so that a performer’s or listener’s motion, heart rate, and breathing are recorded as one data set, a combination the physiology chapter returns to. The costs are careful placement and calibration, drift, and interference from nearby metal and electronics.

An example of a sensor-based motion capture suit used in a performance with the Stavanger Symphony Orchestra in 2023.

The demo below uses a sine wave as a stand-in for the vertical bounce of a body moving to a beat. Its frequency analysis shows a peak near 2 Hz, which corresponds to 120 beats per minute.

Video analysis¶

Ordinary video sits between the qualitative and quantitative approaches described above. Watching and annotating a recording is a qualitative method; the methods in this section instead use computational tools to extract features from the video stream and show them as images and curves, and, at the end of the section, to estimate the positions of body parts the way a marker system does. The first of these, video visualisation, works directly on the pixels. These views can reveal patterns, timing, and coordination in performance or listening that are hard to catch by observation alone. They also bring interpretation and measurement together, which makes them valuable in interdisciplinary research on music-related movement. The video below, from the Music Moves course, introduces video analysis of music-related movement:

(If the video is unavailable, try the archived page.)

From frames to motion¶

A video is a series of still images, called frames, usually 25–60 per second. The simplest way to find motion in it is frame differencing: each frame is subtracted from the one before, pixel by pixel, and only the size of the difference is kept. Pixels that did not change come out black, and pixels where something moved come out bright. The result is a motion image, which shows where the motion is and hides everything that stands still: the background, the furniture, and any part of the body at rest. A small threshold removes the flicker and sensor noise that would otherwise make every pixel change a little.

Two numbers summarise a motion image. The quantity of motion is the share of pixels that changed, a rough measure of how much is moving. The centroid of motion is the average position of those pixels, a rough measure of where the motion happens. Both can be plotted over time like any other signal, and a plot of quantity of motion is often the first thing to look at in a recording of a performance or a dance. They are measures of the image, not of the body: a person moving towards the camera changes fewer pixels than one moving sideways, and a change in the lighting counts as motion too.

Videograms and motiongrams¶

A plot of quantity of motion reduces each frame to one number and loses where in the image things happened. A videogram keeps one dimension of the image. Each frame is collapsed into a single column by averaging the pixels along each row, and the columns are placed side by side over time Jensenius, 2013. Time then runs from left to right and the height of the frame from top to bottom, so a hand moving up and down draws a wave. A motiongram is made in the same way from the motion images, so it shows only what moved. Averaging along each column instead gives a horizontal version, which shows sideways motion.

The figure below shows all of these for a short synthetic video of a hand waving up and down, holding still for a second, and then waving twice as fast.

Figure: Video visualisation of a synthetic video in which a hand waves at 1 Hz, holds still, and waves at 2 Hz. Left: one frame and its motion image. Right: the videogram shows everything in view, including the static table edge at the bottom, while the motiongram and the quantity of motion (QoM) show only the waving hand and drop to zero while it holds still. The quantity of motion also dips at each turning point, where the hand slows down before changing direction.

A motiongram can be read much like a spectrogram, with time along one axis, but with position in the image instead of frequency along the other. A self-similarity matrix takes a different route: it compares every frame with every other frame and colours each pair by how alike they are. Repeated movements then show up as a grid of stripes, and a change of section as a change of texture, just as for audio in machine listening.

Try it live¶

The Video visualiser app does all of this live in the browser with your webcam: the motion image, the quantity and centroid of motion, videograms and motiongrams in both directions, and a self-similarity matrix. The video is analysed on your own computer and is never recorded or sent anywhere. The app builds on VideoViz and the Musical Gestures Toolbox for Python, which offers the same analyses for recorded video, and whose documentation is the place to start if you want to run them yourself (pip install musicalgestures). The older VideoAnalysis application does the same for video files on macOS and Windows without any programming.

Video and sound on one timeline¶

Many questions about music-related movement concern how the movement relates to the sound: whether a nod lands on the beat, whether a bow stroke starts before the note, or how a dancer’s motion follows the music. Answering them needs the video and the audio on the same time axis. The Video scrubber app does this for a recording. It draws a videogram or motiongram of the video above a spectrogram of its sound, and clicking anywhere on either jumps the video to that moment. You can load a video file from your own computer or record a short clip with the camera and microphone; either way, the recording stays in the browser.

Markerless motion capture: from pixels to skeletons¶

The visualisations above work directly on the pixels and say nothing about which pixel belongs to which body part. Markerless (or marker-free) motion capture goes one step further and estimates the pose of the body from ordinary video, or from depth images, a task called pose estimation, using computer vision models trained on large sets of images in which people’s joints have been labelled by hand, an application of the machine learning that machine listening applies to audio. The output is a skeleton: the positions of a few dozen body landmarks, such as wrists, elbows, and the tip of the nose, in every frame. A single camera recovers the skeleton in two dimensions; several cameras, calibrated as for a marker system, can reconstruct it in three. Attaching reflective markers takes time and can interfere with natural movement, especially in dance, ensemble playing, or performances in front of an audience, and markerless tools remove that step. The result is less precise than a marker-based system, and the data must be read with more care, but the data type is the same, positions of body points over time, so the analyses are the same, and the barrier to recording movement in classrooms, concerts, and everyday settings is far lower.

Several open libraries and research pipelines illustrate what is possible today. The list below is not exhaustive, but it shows the main families you are likely to encounter:

Markerless capture still has practical limits. Occlusion (one limb hiding another), loose clothing, uneven lighting, and limited finger detail can reduce accuracy compared with marker-based or sensor-based systems. For many music studies—gesture timing, gross body sway, dance metre, or air performance—they nonetheless offer a practical entry point before investing in lab infrastructure.

Sonification: hearing motion¶

Visualisation turns data into images. Sonification turns data into sound Hermann et al., 2011. Hearing is good at following changes over time and at picking out rhythm and regularity, so a sonification can reveal a periodic pattern or a sudden change that is easy to miss in a plot, and it leaves the eyes free to watch the performer.

The Video sonifier app turns the motiongram straight into sound. Each row of the image controls the level of one frequency band, with the top of the image as the highest frequencies, so motion high in the frame gives a higher, brighter sound and motion near the bottom a lower, darker one. Motion on the left or right of the frame moves the sound to the left or right. When nothing moves, it is silent. In effect, the motiongram the app draws is the spectrogram of the sound it plays.

A critical look: does the right playlist make you run faster?¶

Runners guard their playlists, gyms pump out 130 beats per minute, and music has been called a legal performance-enhancing drug. Race organisers have taken the idea seriously enough to ban headphones in some events. Given what this chapter has said about entrainment and the moving body, the claim is worth measuring rather than assuming.

• The claim is that music makes people exercise harder, longer, or faster, partly by distracting from fatigue and partly by pulling movement into step with the beat.
• The evidence was gathered in a meta-analysis of 139 studies with around 3,600 participants Terry et al., 2020. Music reliably improved mood during exercise, lowered perceived exertion, and improved physical performance measures such as distance covered and time to exhaustion, with a small effect on the efficiency of oxygen use. The performance benefit was moderate at most, was largest for low-to-moderate intensities, and was greater when the music’s tempo was matched to the movement, which is the entrainment mechanism from earlier in this chapter at work.
• The method is mostly short laboratory bouts on treadmills and ergometers with self-selected or prescribed music, compared with silence or noise. Few studies involve trained athletes at competitive intensities.
• The limits follow from that. The reduction in perceived effort fades as intensity rises, since attention narrows to the body once breathing gets hard; the performance effects are averages over many small studies; and mood is measured more consistently than speed.

Music does move the body, and the effect is measurable. But it is modest, it works through attention and mood as much as through the beat, and it is strongest when the exercise is comfortable enough for the music to be heard at all. The playlist helps most on the easy run, not the race.