Video sonifier

Sonification is the auditory counterpart of visualisation: instead of turning data into a picture, it turns them into sound. This app sonifies the motion in front of your camera. Each video frame is compared with the one before, and the difference shows where the image changed. That difference is averaged along every row of the image, which gives one motion value per row, and the column of values is used as the spectrum of a sound. Motion near the top of the image becomes high frequencies, motion near the bottom becomes low frequencies, and when nothing moves there is silence. Press Start, allow camera access, and move.

Privacy: the video is analysed locally in your browser, frame by frame, and immediately discarded. Nothing is recorded, stored, or sent anywhere.

What the camera sees
Camera image
Motion image (frame difference)
The motion image keeps only what changed since the previous frame. Anything that stays still, such as the wall behind you, fades into the background, and a moving hand shows up as an outline. Even when you hold still, a faint speckle of camera noise remains; the noise gate below keeps it from being heard.
Motiongram, heard as a spectrogram The balance meter compares the motion in the left and right halves of the image. It sets where the sound is placed between the left and right speakers, so a sound made by moving on the left comes from the left. Use headphones to hear it clearly.
Shaping the sound
Sensitivity multiplies the motion before it is used, so small movements become louder. The noise gate sets to zero every row whose motion is below the chosen level, which silences camera noise and flicker. The two frequency sliders set which range of the spectrum the image is spread over; a narrow, low range sounds soft and dark, a wide, high range sounds bright and hissy. Smoothing lets each value fade out gradually instead of jumping, which turns quick, buzzy changes into calmer swells. Mirroring makes the image behave like a mirror, so your right hand appears on the right.

Things to try: move one hand slowly up and down in front of the camera and listen to the sound move from a high region to a low one, following your hand in the motiongram. Hold completely still and listen to the silence; then lower the noise gate to zero and hear the camera noise appear. Move only on the left side of the image, then only on the right, with headphones on. Ask a second person to move together with you, one high and one low, and look at the two bands that appear in the motiongram. Finally, narrow the frequency range to 100–400 Hz and compare the sound with 100–4000 Hz.

Based on webvideosonifyer and the Musical Gestures Toolbox.