Translation sketchpad

This sketchpad translates between three data types, a line, a sound, and a text, through one shared contour of pitch and loudness over time. No machine learning model is involved; the mapping runs on fixed rules, written out below, so every step is visible. Everything runs in this browser. Nothing you draw, sing, or record is uploaded anywhere.

Draw
Draw a line, or record your voice, to see a description here.
what you drew    what the microphone heard

Draw a line from left to right with a mouse, a trackpad, or a finger. Height sets pitch across two octaves, from C3 at the bottom to C5 at the top; drawing speed sets loudness. Choosing a scale snaps the pitch to its notes as you draw, so the chromatic scale gives the smoothest glide and the pentatonic scale gives the widest steps. Starting a new line replaces the old one. Drawing needs a pointer; every other control here, including recording your voice, works from the keyboard. The description above is built from a few written rules about the shape of the line; a real captioning model would learn such patterns from examples instead of following rules written by a person.

Play

Play sends the drawn line to a single oscillator: its frequency follows the line's height and its gain follows the drawing speed recorded at each point, over the duration set above. A play head moves across the canvas while it sounds. Loop repeats the drawn line until switched off or the line changes.

Listen and compare

Trace with my voice records up to eight seconds and draws a line from the pitch and loudness it hears; press it again to stop early. Sing what you drew plays your drawn line first and then records you singing it back, so both lines end up on the canvas together. Your browser will ask permission before the microphone is used, and the microphone stops as soon as recording ends. The difference score reported below is a crude measure: it does not account for timing offsets, octave errors, or the difference in tone between an oscillator and a voice.

Draw a line to begin, or try tracing with your voice.

What to notice: a rising line sounds like a rising pitch and nothing else, since height is the only thing this sketchpad reads from your drawing; drawing quickly makes a loud passage, and drawing slowly makes a quiet one. Sing a simple falling phrase, trace it with your voice, and compare its line with one you draw by hand: the two rarely match exactly, which is the point of the difference score.

How it works

The shared representation behind every translation in this app is a contour: an array of pitch and loudness values, sampled fifty times for every second of playback. A drawn line becomes a contour by reading, at each sample position along its width, the pitch that its height maps to (quantised to the chosen scale) and the loudness recorded from how fast the pointer was moving when it passed that point. A recording becomes a contour by splitting the audio into overlapping 2048-sample frames, roughly forty-six milliseconds each, and running autocorrelation on every frame: the frame is compared against time-shifted copies of itself, the shift that matches best gives the pitch, and a frame is only called "voiced" and given a pitch when that match is strong enough to clear a confidence threshold. The root-mean-square level of each frame gives its loudness. Playing a contour back sends its pitch and loudness arrays straight to a single oscillator's frequency and gain, so the sound never has to first become anything else. The text description is built from three simple rules: whether the pitch rises, falls, or holds across each third of the contour, and which third is loudest. The pitch tracker is a plain autocorrelation with no correction for octave errors, so it can report a pitch exactly one octave from the true one; this is the main reason the difference score can mislead.

This app bundles no data of its own and calls no external service.