Skip to content

MGT-python: Musical Gestures Toolbox

PyPi version GitHub license CI Documentation

The Musical Gestures Toolbox for Python (musicalgestures) is a collection of tools for visualising and analysing motion in video recordings, together with the sound that accompanies them. It was developed for research on music-related body motion, but it works on any video or audio file, and it supports qualitative and quantitative approaches on equal terms: images made for human looking, and measurements made for further computation.

MGT python demo

Two doors in

  • Gallery — every kind of picture the toolbox makes, each linking to the page that makes it. Start here if you know what you want an analysis to look like but not what it is called.
  • Concepts — the ideas the toolbox is organised around: motion, action, and gesture; position, posture, and pose; quantity of motion and its limits; visualisation as a way of looking; the effort layer. Start here if you want the function names to make sense before you call them.

Where to start, by field

  • Music researchers and students: the Gallery shows the analyses at a glance; Concepts connects them to the musical-gestures literature; the Quick Start gets a first motiongram out of your own video in minutes.
  • Psychologists: Concepts maps posture and pose onto your literature's terms; pose tracking turns video into codeable trajectories; the annotation and co-occurrence tools (see the API reference) carry the step from video to coded units.
  • Human movement scientists: pose tracking gives landmark trajectories from plain video; the sound–movement toolkit holds the array-level functions, postural sway included; Concepts states exactly what is meant by position and posture here.
  • Computer scientists: the API reference and core classes describe the machinery; contributing and testing describe the workshop.

Quick start

pip install musicalgestures

The package installs its Python dependencies automatically. Install ffmpeg separately to enable video processing; see the installation guide.

import musicalgestures as mg

# Load a video
v = mg.MgVideo('dance.mp4')

# Create visualisations — call .show() to display the result
v.motiongrams().show()
v.average().show()

# Motion and audio analysis
v.motion().show()
v.audio.spectrogram().show()

Analysis methods return result objects (MgVideo, MgImage, or MgFigure) and do not auto-render; .show() displays them.

Open In Colab

What the toolbox does

  • Video analysis: motion detection, optical flow, motion vectors, motion tempo, Eulerian video magnification, and motion descriptors (energy, smoothness, entropy, spectral)
  • Visualisations: motiongrams, videograms, motion history, heatmaps, and space-time displays (stroboscope, silhouette waterfall, 3D space-time volume)
  • Pose estimation: MediaPipe (default), YOLO, RTMPose and OpenPose backends on one trajectory contract, with trajectory summaries, motion trails, and per-segment statistics
  • Segmentation: actions cut from motion envelopes, postures cut from landmark trajectories, pulse and cycle segmentation, long-video segmentation, and person tracking
  • Audio analysis: waveforms, spectrograms, MFCC, chromagrams, tempo and beat tracking, and sonomotiongrams (motion turned into sound)
  • Audio–motion analysis: tempo similarity, phase synchrony, structural similarity, and per-body-part motion–audio coupling for a single performer
  • Sound–movement research toolkit: lower-level, array-based functions for pulse and cycle segmentation, cross-modal alignment, quantity of motion, postural sway, physiology, and motion-capture I/O—see the Sound–Movement Analysis Toolkit
  • 360 video: projection handling, per-direction views, the anglegram, and audio-energy-map overlays—see Video Analysis
  • Integration: works with the NumPy, SciPy, librosa, and Matplotlib ecosystems, on Linux, macOS, and Windows

Learning path

The wiki is the course: numbered chapters that read in order, basics first, with exercises on the bundled example videos. This site is the reference: it answers "how do I do X" and "what exactly does this return". The two link to each other throughout.

Runtime behaviour

  • pose() defaults to the MediaPipe backend and downloads its weights on demand; the OpenPose models download their larger Caffe weights on first use instead.
  • In notebook and batch execution, pose weight downloads are attempted automatically instead of prompting for stdin.
  • If CUDA-backed OpenCV DNN support is unavailable, pose(device='gpu') runs on the CPU instead (switching to the MediaPipe backend when it is installed).
  • flow.dense(), flow.sparse(), and blur_faces() run on CPU by default (use_gpu=False); pass use_gpu=True to attempt CUDA acceleration with automatic CPU fallback.
  • get_cuda_device_count() can be used to check CUDA visibility from OpenCV.

Four toolboxes come out of the fourMs lab at the University of Oslo. They are separate packages with separate release cycles, but they are built to be used together and share several implementations, so a measure computed in one agrees with the same measure computed in another.

  • ambiscape—soundscapes: the sonic ambience of a place, across level, spectral, spatial, temporal, ecological and source descriptors
  • musiscape—music collections: comparing many tracks and albums held as audio files in folders
  • micromotion—human micromotion: quantity of motion from optical markers, accelerometers, respiration belts and force plates

MGT-python and ambiscape divide the work cleanly: MGT owns the pixels (motion analysis, pose, 360 handling, video visualisation), while ambiscape owns the samples (soundscape levels, spatial audio, sound-event taxonomies). MGT's audio functions cover quick looks; for serious soundscape work, install the bridge—pip install "musicalgestures[soundscape]"—and pull ambiscape's session features straight into MgFeatures on a shared wall-clock time base (see musicalgestures._soundscape and musicalgestures._timecode).

Academic background

This toolbox builds on the Musical Gestures Toolbox for Matlab, which again builds on the Musical Gestures Toolbox for Max. The software is maintained by the fourMs lab at RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, University of Oslo.

Support and community

Citing

If you use MGT-python in your research, please cite the article and the software:

The Zenodo DOI above is the concept DOI and always resolves to the newest version; where the exact behaviour matters, cite the version-specific DOI of the release you ran as well (listed on the Zenodo record).

@inproceedings{laczkoReflectionsDevelopmentMusical2021,
    title = {Reflections on the Development of the Musical Gestures Toolbox for Python},
    author = {Laczkó, Bálint and Jensenius, Alexander Refsum},
    booktitle = {Proceedings of the Nordic Sound and Music Computing Conference},
    year = {2021},
    address = {Copenhagen},
    url = {http://urn.nb.no/URN:NBN:no-91935}
}

License

MGT-python is released under the GNU General Public License v3 (GPLv3).