Sensor-to-sound mapper

This app lets you design your own action–sound mapping: you choose which sensor axis drives which sound parameter, then play with the result. On a phone, the axes come from its motion sensors; on a laptop, they come from the mouse or trackpad. Everything runs in this page. No sensor reading or sound is recorded, stored, or sent anywhere; it only ever drives the sound you hear while the page is open.

Sensors

On a laptop, move the mouse or a trackpad pointer over this page to drive the pointer axes below; no button is needed for those. On a phone or tablet, press “Enable motion sensors” to add the acceleration and rotation axes; the browser will ask permission first.

Sound
Mapping

Each row below sends one sensor axis to one sound parameter. Choose the source axis, set the input range that the axis is expected to move within, invert the direction if you want, and smooth out jitter. The sensible mapping below stands in for shake with a single rotation rate axis (β), or with the pointer’s Y axis on a laptop with no motion sensors.

One row per sound parameter, with its source axis and calibration
Parameter Source axis Input minimum Input maximum Invert Smoothing
Learnability experiment

Task: make the sound go up, then get brighter. “Up” means raising the pitch noticeably above where it started, and “brighter” means raising the filter cutoff noticeably above where it started once the pitch has risen; if a parameter starts the stage already near its top, the task asks you to bring it down first so that a rise is still possible. Design your own mapping above, run the task and note the time, then press “Randomise mapping” and try the same task again without looking at the table.

Time: 0.0 s
Move the mouse, or enable motion sensors, then turn the sound on to begin.

A mapping that sends one axis straight to one parameter is a direct mapping; a mapping that mixes several axes into one parameter, or first passes the reading through some maths, is indirect and harder to feel by ear. A one-to-one mapping, where the same action always moves the same parameter by the same amount, is learnable because practice has something consistent to improve; a many-to-many mapping, where several axes cross-affect several parameters at once, gives the body nothing stable to aim at. This mapping table is the interface layer of the five layers this book uses (data, model, interface, practice and culture): it sits between the raw sensor readings and the practice of learning to play them.

How it works

Every animation frame, the app reads the current value of each available axis: acceleration and rotation rate from the phone’s motion sensors, or pointer position and speed from the mouse or trackpad. For each sound parameter, the axis chosen in the mapping table is rescaled from the input minimum and maximum you set to a range from 0 to 1, inverted if you asked for that, then smoothed so that a jump in the raw reading arrives as a gradual change rather than a jump in sound. That smoothed value drives an oscillator’s pitch, a filter’s cutoff frequency for brightness, and the speed of a pulsing gain for tempo. The learnability task compares your own mapping against a randomised one by timing how long each takes to complete. Shake has no sensor of its own, so the sensible mapping approximates it with a single rotation rate axis (β); on a laptop with no motion sensors, the pointer’s Y axis stands in for it instead. This app bundles no data files, so there is no licence to name, and it makes no network requests.