Skip to content

Motion vectors

Every inter-frame codec has already searched for the displacement of each macroblock in order to compress the video. These two functions read that work back out instead of doing it again: motionvectors() draws it, motionvectordata() returns it as numbers.

Drawing them

motionvectors() renders the vectors as arrows over the video, via ffmpeg's codecview filter. It is a decoder-level view of motion with nothing recomputed, and it needs only ffmpeg.

Only codecs that carry motion vectors will show anything. Intra-only formats—MJPEG, common in .avi files—have none.

Reading them as data

motionvectordata() returns one row per decoded frame: the time, the picture type, how many vectors the frame carried, an area-weighted magnitude, and the typical displacement median_dx / median_dy of the blocks that moved.

It is cheap in a way that differencing pixels is not. On a 103-minute recording, decoding took 2.2 s and decoding with the vectors took 2.7 s, against 27 minutes to compute frame-differenced quantity of motion over the same file.

Filter by picture type. This is the difference between a usable signal and a poor one. Measured against exact quantity of motion on a 100-minute corpus:

frames used r
P-frames only 0.87
each type standardised, then pooled 0.70
everything pooled 0.54

B-frames are about three quarters of the frames, and their vectors point both ways over varying temporal distances, so they are not measuring the same quantity. Keeping only picture_type == 'P' leaves roughly 12.6 Hz on 50 fps footage, which is ample for anything at the scale of a gesture.

It measures the encoder, not the scene. An encoder may choose any vector that predicts a block cheaply, so on flat or still regions the vectors mean little. Across 18 windows of six recordings the correlation with quantity of motion had a median of 0.82—but 0.86 in windows with motion in them, against 0.57 in windows without. A re-encoded proxy carries its proxy encoder's decisions, not the camera's.

source gives the direction of the reference, never the distance. It is plus or minus one even when the encoder predicted from four frames back, so multi-frame predictions read as multiples of the true per-frame displacement. On a block moving 4 pixels per frame, 53 per cent of vectors came back as 4 and the rest as 8 or 16. The medians are robust to this; magnitude, being a sum, is not, and is best treated as a quantity to correlate against itself over time rather than as pixels per second.

Looking at a lot of material quickly

motionvectoroverview() is the one to reach for first on unfamiliar video. It produces one sheet per recording from a single decode, carrying the three kinds of representation a first pass wants:

  • spatial—the area motion covered, coloured by direction, and again as plain amount
  • temporal—motion across the whole recording, the curve to scrub against
  • spatio-temporal—horizontal and x-motiongrams, where a body crossing the room draws a diagonal

Motion-vector overview sheet of the bundled dancer example

The bundled dancer.avi on one sheet. The amount map draws the arc the arms swept around a body that never moved its feet, and the motiongrams carry the same dance in the classic orientations: the y-motiongram runs time rightward and holds the arm oscillations, since this dance is vertical movement, and the x-motiongram runs time downward with the room's width across.

About five minutes for a 100-minute 1920x1080 recording, against roughly twenty-four for the same views computed one at a time, because decoding is nearly the whole cost and each separate view opens the file again. Every array is on the returned figure's .data, so the same pass can feed further analysis without decoding twice.

In the direction panel, hue is the bearing, value is how much motion there was, and saturation is directional coherence—how consistently that part of the room moved one way. Grey means "moved, but not in any one direction", which is the honest description of most of a room during improvisation. Coherence is under-sampled above about 1.5 Hz on 50 fps footage, since P-frames arrive at roughly a quarter of the frame rate: right for a body crossing a room, wrong for a shaking hand.

Every timescale at once

motionscape() draws the quantity of motion at every timescale at once: a triangle whose base row is the motion in short windows, each row above a longer window, and the apex one window covering the whole recording. The construction comes from Craig Sapp's keyscapes, which do the same for tonality.

Motionscape of the bundled dancer example

The bundled dancer.avi: bright columns at the base are single bursts, the dark mass around 0.7--0.8 minutes is a stretch of stillness long enough to survive widening windows, and the arches show where neighbouring episodes merge.

It answers a question the flat curve cannot: at what scale does this recording stop looking like one thing? Evenly continuous material is flat all the way up. A few busy patches separated by quiet stay separate at the base and merge into one mass higher up, and the height at which they merge is the length of the structure joining them.

It is a real triangle: each row is only as wide as the number of places a window that long can sit, and the narrowing is the information rather than a cosmetic frame.

Nothing in it assumes motion vectors. Pass track= any per-frame series—mg_motion's QomRaw, or a qom.f4 read off disk—to scape a frame-differenced quantity of motion instead. Built from a cumulative sum, so the cost does not grow with window length.

Reproducibility

Frame-threaded decoding drops motion vectors, and not the same ones twice. Three identical runs over one clip returned 28,702, 29,649 and 29,975 vectors with threading on, and 29,975 every time with it off.

On a real 1920x1080 recording the loss is 222 vectors in 25.7 million—0.0009 per cent, far below anything that changes a reading—and threading is about 4.5 times faster, so it is the default. Pass deterministic=True to anything in this module when a result has to reproduce exactly.

Which files this works on, measured

One 120 s window of real dancing, re-encoded every way, comparing the vectors against the frame difference of that same file, and timing both.

Codec decides whether it works at all. ffmpeg exports motion vectors for H.264 and MPEG-4 Part 2 and, in this build, nothing else—HEVC and VP9 decode perfectly and return zero vectors, so the result is all zeros with no error. motionvectordata warns when a file carries none, because zeros otherwise read as stillness.

codec r (P-frames) vectors
H.264 0.82 yes
MPEG-4 Part 2 0.72 yes
HEVC — none exported
VP9 — none exported
MJPEG — none (intra-only)

Resolution barely matters, which is the useful part. The vectors are computed on macroblocks, so the cost scales with area but the agreement does not.

width r (P-frames) vector cost framediff cost speed-up
320 0.73 0.12 s 1.11 s 9×
640 0.82 0.24 s 2.21 s 9×
1280 0.80 0.88 s 5.60 s 6×
1920 0.81 1.97 s 11.57 s 6×

Frame rate helps. More P-frames means more of the signal, and at 50 fps the agreement with the source recording's own motion is much the better for it.

fps r (P-frames) r against the 1920 source
12 0.72 0.50
25 0.82 0.45
50 0.89 0.72

Compression quality hardly matters, and heavier compression is if anything better.

CRF r (P-frames)
18 0.75
23 0.82
30 0.84
38 0.81

At high quality the encoder subdivides finely and spends vectors on detail that is not motion; at low quality it keeps large blocks that track the body. So a proxy is not a handicap here, which is worth knowing given how often analysis runs on one.

Across the whole sweep the vectors ran 5 to 25 times faster than differencing the same file, and pooled correlation stayed between 0.01 and 0.53 against 0.72--0.89 on P-frames alone—the single most important thing to get right.

Reading the vectors needs PyAV—pip install musicalgestures[motionvectors]. ffprobe will say the side data is present but will not print it, and ffmpeg has no numeric dump, so there is no command-line route. Drawing them is unaffected.

MgMotionVectorData dataclass

MgMotionVectorData(time, picture_type, n_vectors, magnitude, median_dx, median_dy)

One row per decoded frame, as arrays.

picture_type is here because it decides whether the rest is usable. Measured on a 100-minute corpus against exact frame-differenced quantity of motion, the magnitude below correlates at r = 0.87 on P-frames alone and r = 0.54 once B-frames are pooled in, so a caller who ignores this field gets a much worse signal than the codec is offering. Filtering to picture_type == 'P' costs three quarters of the frames and is usually the right trade.

MgMotionVectorViews dataclass

MgMotionVectorViews(history_weight, history_vx, history_vy, history_coherence, time, magnitude, motiongram_y, motiongram_x)

Every motion-vector view of one recording, from a single decode.

Three kinds of representation, which is what a first pass over unfamiliar material wants: spatial (where motion happened, and which way), temporal (how much, over time), and spatio-temporal (the motiongrams, position against time, in the classic orientations: motiongram_horizontal as (rows, frames) with time rightward, motiongram_vertical as (frames, cols) with time downward).

The temporal view is an envelope, not a track. magnitude against time is the motion envelope at the encode's own reference cadence --- about 12.6 Hz on a 50 fps B-frame encode, not 50 Hz --- for roughly a tenth of a full decode's cost. Measured against the exact frame-difference quantity of motion on a six-recording dance corpus, it correlates at median r = 0.82 across windows, falling to 0.57 where little moves: the envelope is for finding structure, not for measuring stillness. B-frames are excluded by default (p_frames_only=True) because their future-referencing vectors pool badly; including them costs about 0.3 of r.

mg_motionvectors

mg_motionvectors(self, target_name=None, overwrite=True)

Renders a video visualising the motion vectors encoded in the input video.

Inter-frame codecs (MPEG-1/2/4, H.264, H.265, …) store motion vectors that describe how macroblocks move between frames. This method uses FFmpeg's codecview filter (with -flags2 +export_mvs) to draw those vectors as arrows on top of the video, giving a quick, decoder-level view of motion without any re-computation.

NB: Only codecs that actually carry motion vectors will show arrows. Intra-only formats (e.g. MJPEG, common in .avi files) have none — convert to an inter-frame codec first (e.g. via show(mode='notebook') which makes an mp4, or any mp4/h264 source) to see motion vectors.

Parameters:

Name Type Description Default
target_name str

Target output name for the video. Defaults to None (which uses the input filename with the suffix "_motionvectors").

None
overwrite bool

Whether to allow overwriting existing files or to automatically increment the target filename. Defaults to True.

True

Returns:

Name Type Description
MgVideo MgVideo

An MgVideo pointing to the rendered motion-vector video.

Source code in musicalgestures/_motionvectors.py
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def mg_motionvectors(self: "musicalgestures.MgVideo", target_name=None, overwrite=True) -> "musicalgestures.MgVideo":
    """
    Renders a video visualising the motion vectors encoded in the input video.

    Inter-frame codecs (MPEG-1/2/4, H.264, H.265, …) store motion vectors that describe
    how macroblocks move between frames. This method uses FFmpeg's ``codecview`` filter
    (with ``-flags2 +export_mvs``) to draw those vectors as arrows on top of the video,
    giving a quick, decoder-level view of motion without any re-computation.

    NB: Only codecs that actually carry motion vectors will show arrows. Intra-only
    formats (e.g. MJPEG, common in ``.avi`` files) have none — convert to an inter-frame
    codec first (e.g. via ``show(mode='notebook')`` which makes an mp4, or any mp4/h264
    source) to see motion vectors.

    Args:
        target_name (str, optional): Target output name for the video. Defaults to None
            (which uses the input filename with the suffix "_motionvectors").
        overwrite (bool, optional): Whether to allow overwriting existing files or to
            automatically increment the target filename. Defaults to True.

    Returns:
        MgVideo: An MgVideo pointing to the rendered motion-vector video.
    """
    of, fex = os.path.splitext(self.filename)

    target_name = resolve_filename(of, '_motionvectors' + fex, target_name, overwrite)

    # -flags2 +export_mvs must precede -i so the decoder exports motion vectors;
    # codecview then draws them (pf=P-frame forward, bf/bb=B-frame forward/backward).
    cmd = [
        'ffmpeg', '-y', '-flags2', '+export_mvs', '-i', self.filename,
        '-vf', 'codecview=mv=pf+bf+bb', '-q:v', '3', target_name,
    ]
    ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering motion vectors:')

    self.motionvectors_video = musicalgestures.MgVideo(
        target_name, color=self.color, returned_by_process=True)
    return self.motionvectors_video

mg_motionvectordata

mg_motionvectordata(self)

Read the motion vectors the codec already computed, as numbers rather than arrows.

motionvectors() renders these as a video to look at. This returns them as data, and the difference in cost is the point: an inter-frame codec has already searched for the displacement of every macroblock, so reading its answer is close to free, while differencing pixels does the same search again in Python. On a 103-minute recording, decoding took 2.2 s and decoding with the vectors took 2.7 s, against 27 minutes for the frame-differenced quantity of motion.

What it is not. The vectors are the encoder's decisions, not a measurement of the scene: an encoder is entitled to any vector that predicts the block cheaply, and on flat or still regions it will pick one that has nothing to do with motion. The result tracks quantity of motion well where there is motion to track and poorly where there is not --- on that same corpus, median r = 0.86 in windows with motion and 0.57 in windows without. Re-encoded proxies carry their proxy encoder's vectors, not the camera's.

Vectors are normalised to point forward in time. A B-frame may predict a block from a later frame, which reverses the sign; dividing by the vector's source corrects that, so a block moving right reads as positive whichever way it was predicted. Without it a B-frame's vectors point backwards half the time and averaging them gives roughly nothing.

The reference distance is corrected from the cadence, with two residuals. source records only the direction, plus or minus one, never how many frames away the reference was, so the distance is recovered instead from the picture-type cadence: past-referencing vectors are divided by the display-order gap to the previous reference frame, which makes a P-frame after a run of B-frames read per-frame displacement rather than the whole span's. What the cadence cannot see: blocks reaching an OLDER reference through multi-reference prediction --- measured on a block moving 4 pixels per frame, such blocks read as 8 or 16 with source at plus or minus one throughout --- and B-frame vectors pointing at a future reference, which default filtering excludes. So median_dx and median_dy are medians for a reason and are robust to the residuals; magnitude is a sum and inherits what remains of the over-count. Treat magnitude as a quantity to correlate against itself over time, which is what it was validated for, rather than as pixels per second.

Returns:

Name Type Description
MgMotionVectorData MgMotionVectorData

time in seconds, picture_type as 'I'/'P'/'B',

MgMotionVectorData

n_vectors, magnitude (the area-weighted sum of displacement, which is the

MgMotionVectorData

quantity to compare against quantity of motion), and median_dx / median_dy

MgMotionVectorData

(the typical displacement in pixels of the blocks that moved --- taken over

MgMotionVectorData

every block instead, these would report the still background's zero however fast

MgMotionVectorData

the one moving thing was going). Frames with no vectors, which includes every

MgMotionVectorData

intra frame, read as zero throughout.

Raises:

Type Description
ImportError

if PyAV is not installed. ffprobe reports that the side data exists but will not print the vectors themselves, and ffmpeg has no numeric dump, so there is no route to these numbers through the command line.

Source code in musicalgestures/_motionvectors.py
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
def mg_motionvectordata(self: "musicalgestures.MgVideo") -> "MgMotionVectorData":
    """Read the motion vectors the codec already computed, as numbers rather than arrows.

    `motionvectors()` renders these as a video to look at. This returns them as data, and
    the difference in cost is the point: an inter-frame codec has already searched for the
    displacement of every macroblock, so reading its answer is close to free, while
    differencing pixels does the same search again in Python. On a 103-minute recording,
    decoding took 2.2 s and decoding with the vectors took 2.7 s, against 27 minutes for
    the frame-differenced quantity of motion.

    **What it is not.** The vectors are the encoder's decisions, not a measurement of the
    scene: an encoder is entitled to any vector that predicts the block cheaply, and on
    flat or still regions it will pick one that has nothing to do with motion. The
    result tracks quantity of motion well where there is motion to track and poorly where
    there is not --- on that same corpus, median r = 0.86 in windows with motion and
    0.57 in windows without. Re-encoded proxies carry their proxy encoder's vectors, not
    the camera's.

    **Vectors are normalised to point forward in time.** A B-frame may predict a block
    from a later frame, which reverses the sign; dividing by the vector's `source`
    corrects that, so a block moving right reads as positive whichever way it was
    predicted. Without it a B-frame's vectors point backwards half the time and averaging
    them gives roughly nothing.

    **The reference distance is corrected from the cadence, with two residuals.**
    `source` records only the direction, plus or minus one, never how many frames away
    the reference was, so the distance is recovered instead from the picture-type
    cadence: past-referencing vectors are divided by the display-order gap to the
    previous reference frame, which makes a P-frame after a run of B-frames read
    per-frame displacement rather than the whole span's. What the cadence cannot see:
    blocks reaching an OLDER reference through multi-reference prediction --- measured
    on a block moving 4 pixels per frame, such blocks read as 8 or 16 with `source` at
    plus or minus one throughout --- and B-frame vectors pointing at a future
    reference, which default filtering excludes. So `median_dx` and `median_dy` are
    medians for a reason and are robust to the residuals; `magnitude` is a sum and
    inherits what remains of the over-count. Treat `magnitude` as a quantity to
    correlate against itself over time, which is what it was validated for, rather
    than as pixels per second.

    Returns:
        MgMotionVectorData: `time` in seconds, `picture_type` as 'I'/'P'/'B',
        `n_vectors`, `magnitude` (the area-weighted sum of displacement, which is the
        quantity to compare against quantity of motion), and `median_dx` / `median_dy`
        (the typical displacement in pixels **of the blocks that moved** --- taken over
        every block instead, these would report the still background's zero however fast
        the one moving thing was going). Frames with no vectors, which includes every
        intra frame, read as zero throughout.

    Raises:
        ImportError: if PyAV is not installed. ffprobe reports that the side data exists
            but will not print the vectors themselves, and ffmpeg has no numeric dump, so
            there is no route to these numbers through the command line.
    """
    try:
        import av
    except ImportError as exc:                       # pragma: no cover - trivial branch
        raise ImportError(
            "Reading motion vectors as data needs PyAV. Install it with "
            "`pip install musicalgestures[motionvectors]`, or `pip install av`. "
            "The rendering version, MgVideo.motionvectors(), needs only ffmpeg."
        ) from exc
    import numpy as np

    container = av.open(self.filename)
    stream = container.streams.video[0]
    stream.thread_type = "AUTO"
    #: AV_CODEC_FLAG2_EXPORT_MVS. PyAV exposes flags2 as a plain int and does not
    #: re-export the constant, so it is written out rather than imported.
    stream.codec_context.flags2 |= 1 << 28

    time: list[float] = []
    kinds: list[str] = []
    counts: list[int] = []
    magnitude: list[float] = []
    med_dx: list[float] = []
    med_dy: list[float] = []
    #: The reference distance, recovered from the cadence. `source` carries only the
    #: SIGN of the reference --- measured on a real encode it is plus or minus one
    #: throughout, whatever the distance --- so a P-frame that follows a run of
    #: B-frames reports its whole multi-frame displacement as one frame's. What IS
    #: recoverable while decoding in display order is the gap back to the previous
    #: reference frame (I or P), and dividing the past-referencing vectors by that gap
    #: corrects the cadence part exactly. Two residuals remain and are documented
    #: rather than hidden: blocks reaching an OLDER reference through multi-reference
    #: prediction, and B-frame vectors pointing at a future reference --- the toolbox
    #: excludes B-frames by default, and their vectors were already the worse signal.
    display_idx = -1
    last_ref_idx = None
    for frame in container.decode(stream):
        display_idx += 1
        kind = _PICTURE_TYPES.get(int(frame.pict_type), "?")
        past_gap = 1.0 if last_ref_idx is None else max(1.0, display_idx - last_ref_idx)
        if kind in ("I", "P"):
            last_ref_idx = display_idx
        time.append(float(frame.pts * stream.time_base) if frame.pts is not None
                    else (time[-1] if time else 0.0))
        kinds.append(kind)
        vectors = frame.side_data.get("MOTION_VECTORS")
        table = vectors.to_ndarray() if vectors is not None else None
        if table is None or len(table) == 0:
            counts.append(0)
            magnitude.append(0.0)
            med_dx.append(0.0)
            med_dy.append(0.0)
            continue
        scale = np.maximum(table["motion_scale"].astype(np.float64), 1)
        #: Two corrections in one division: ffmpeg defines the vector as
        #: `src = dst + motion / motion_scale`, so it points from where the block is
        #: now back to where it came from --- content travelling right carries a
        #: NEGATIVE motion_x --- and `source` is negative for a past reference,
        #: positive for a future one. Dividing by `source` undoes both senses; the
        #: cadence gap above then scales the past-referencing multi-frame predictions
        #: down to one frame.
        source = table["source"].astype(np.float64)
        source[source == 0] = -1
        dx = table["motion_x"].astype(np.float64) / scale / source
        dy = table["motion_y"].astype(np.float64) / scale / source
        past = source < 0
        dx[past] /= past_gap
        dy[past] /= past_gap
        area = table["w"].astype(np.float64) * table["h"]
        counts.append(len(table))
        magnitude.append(float((np.hypot(dx, dy) * area).sum()))
        #: Over the blocks that moved, not over every block. Most of a frame is
        #: usually still, so a median over all blocks is the background's zero however
        #: fast the one moving thing is going, which is not a useful answer to "which
        #: way did it move". Threshold-free: a block counts as moving if the encoder
        #: gave it a displacement at all.
        moving = (dx != 0) | (dy != 0)
        med_dx.append(float(np.median(dx[moving])) if moving.any() else 0.0)
        med_dy.append(float(np.median(dy[moving])) if moving.any() else 0.0)
    codec = stream.codec_context.name
    container.close()

    if counts and not any(counts):
        #: An array of zeros means "nobody moved" and "this codec carries no vectors"
        #: equally well, and the caller cannot tell them apart. ffmpeg exports vectors
        #: for H.264 and MPEG-4 Part 2; HEVC and VP9 decode fine and return none.
        warnings.warn(
            f"No motion vectors in this file: the {codec} stream decoded but carried "
            f"none, so every value below is zero. ffmpeg exports motion vectors for "
            f"H.264 and MPEG-4 Part 2, and not for HEVC, VP9 or any intra-only format. "
            f"Re-encode to H.264 to read them.",
            UserWarning, stacklevel=2)

    return MgMotionVectorData(
        time=np.asarray(time, dtype=np.float64),
        picture_type=np.asarray(kinds),
        n_vectors=np.asarray(counts, dtype=np.int64),
        magnitude=np.asarray(magnitude, dtype=np.float64),
        median_dx=np.asarray(med_dx, dtype=np.float64),
        median_dy=np.asarray(med_dy, dtype=np.float64),
    )

motion_vector_grid

motion_vector_grid(filename, deterministic=False, threshold=0.0)

Per-frame displacement fields at macroblock resolution, yielded one at a time.

Everything spatial in this module is built on this: the history image sums it over time, the motiongrams reduce it along one axis per frame, the waterfall bins those. Decoding is the expensive part, so it happens once per view rather than once per frame of interest.

A generator, and that is not a style choice. The first version returned the frames stacked into arrays of shape (frames, rows, cols). On a 100-minute 1920x1080 recording that is 310,368 frames over a 68 by 120 grid --- 20 GB per array, and it built three. It ran on every clip in the test suite and would have exhausted memory on the first real recording. Consumers now accumulate as they go, and nothing holds more than one frame's grid.

The grid is the codec's own macroblock lattice --- typically 16 pixels, sometimes 8 or 4 within the same frame when the encoder subdivides. Vectors are painted into the cells their block covers, so a 16x16 block contributes to all of its cells and the result does not favour finely-subdivided regions for having more vectors.

Parameters:

Name Type Description Default
threshold float

Drop vectors shorter than this many pixels. The counterpart of mg_motion's threshold, in the units a displacement actually has. H.264 codes to quarter-pel and the encoder spends short vectors on rate rather than on motion: measured on a dance corpus, 46 per cent of vectors are exactly zero and the median non-zero one is 0.79 px, while real limb motion runs to tens of pixels. Defaults to 0.0, which keeps everything, so no existing caller's numbers change silently.

0.0

Yields:

Name Type Description
tuple

(vx, vy, weight, time, is_p) per frame, each of vx/vy/weight

shaped (rows, cols). vx/vy are mean displacement per cell in pixels,

weight the moved area, time in seconds, is_p whether it is a P-frame.

Source code in musicalgestures/_motionvectors.py
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
def motion_vector_grid(filename, deterministic=False, threshold=0.0):
    """Per-frame displacement fields at macroblock resolution, **yielded one at a time**.

    Everything spatial in this module is built on this: the history image sums it over
    time, the motiongrams reduce it along one axis per frame, the waterfall bins those.
    Decoding is the expensive part, so it happens once per view rather than once per
    frame of interest.

    **A generator, and that is not a style choice.** The first version returned the
    frames stacked into arrays of shape (frames, rows, cols). On a 100-minute 1920x1080
    recording that is 310,368 frames over a 68 by 120 grid --- 20 GB per array, and it
    built three. It ran on every clip in the test suite and would have exhausted memory
    on the first real recording. Consumers now accumulate as they go, and nothing holds
    more than one frame's grid.

    The grid is the codec's own macroblock lattice --- typically 16 pixels, sometimes 8
    or 4 within the same frame when the encoder subdivides. Vectors are painted into the
    cells their block covers, so a 16x16 block contributes to all of its cells and the
    result does not favour finely-subdivided regions for having more vectors.

    Args:
        threshold (float, optional): Drop vectors shorter than this many pixels. The
            counterpart of `mg_motion`'s `threshold`, in the units a displacement actually
            has. H.264 codes to quarter-pel and the encoder spends short vectors on rate
            rather than on motion: measured on a dance corpus, 46 per cent of vectors
            are exactly zero and the median non-zero one is 0.79 px, while real limb
            motion runs to tens of pixels. Defaults to 0.0, which keeps everything, so no
            existing caller's numbers change silently.

    Yields:
        tuple: `(vx, vy, weight, time, is_p)` per frame, each of `vx`/`vy`/`weight`
        shaped (rows, cols). `vx`/`vy` are mean displacement per cell in pixels,
        `weight` the moved area, `time` in seconds, `is_p` whether it is a P-frame.
    """
    try:
        import av
    except ImportError as exc:                       # pragma: no cover - trivial branch
        raise ImportError(
            "Reading motion vectors as data needs PyAV. Install it with "
            "`pip install musicalgestures[motionvectors]`, or `pip install av`."
        ) from exc
    import numpy as np

    container = av.open(filename)
    stream = container.streams.video[0]
    #: FRAME-THREADED DECODING DROPS MOTION VECTORS, AND NOT THE SAME ONES TWICE.
    #: Measured on a synthetic clip, three identical runs returned 28,702, 29,649 and
    #: 29,975 vectors with threading on, and 29,975 every time with it off. On a real
    #: 1920x1080 recording the loss is 222 vectors in 25.7 million --- 0.0009 per cent,
    #: far below anything that changes a result --- but it is still not reproducible,
    #: and threading is 4.5x faster (5 minutes a session against 23). So the fast,
    #: slightly lossy path is the default for looking at material, and `deterministic`
    #: is there for anything that has to give the same answer twice.
    stream.thread_type = "NONE" if deterministic else "AUTO"
    stream.codec_context.flags2 |= 1 << 28
    #: The reconstructed picture is never looked at here --- only the side data is --- so
    #: the two most expensive reconstruction stages can be skipped. Measured at
    #: 1,279 fps against 861 on 1920x1080, with the vector count unchanged.
    try:
        stream.codec_context.options = {"skip_loop_filter": "all", "skip_idct": "all"}
    except Exception:                                # pragma: no cover - older PyAV
        pass
    width = stream.codec_context.width
    height = stream.codec_context.height
    cols = max(1, -(-width // _GRID))
    rows = max(1, -(-height // _GRID))

    previous_time = 0.0
    display_idx = -1
    last_ref_idx = None
    try:
        for frame in container.decode(stream):
            display_idx += 1
            kind = _PICTURE_TYPES.get(int(frame.pict_type), "?")
            #: The cadence gap to the previous reference frame; see motionvectordata
            #: for why `source` cannot provide the distance itself.
            past_gap = (1.0 if last_ref_idx is None
                        else max(1.0, display_idx - last_ref_idx))
            if kind in ("I", "P"):
                last_ref_idx = display_idx
            vx = np.zeros((rows, cols), np.float64)
            vy = np.zeros((rows, cols), np.float64)
            w = np.zeros((rows, cols), np.float64)
            t = (float(frame.pts * stream.time_base)
                 if frame.pts is not None else previous_time)
            previous_time = t
            is_p = kind == "P"
            mvs = frame.side_data.get("MOTION_VECTORS")
            table = mvs.to_ndarray() if mvs is not None else None
            if table is not None and len(table):
                scale = np.maximum(table["motion_scale"].astype(np.float64), 1)
                source = table["source"].astype(np.float64)
                source[source == 0] = -1
                dx = table["motion_x"].astype(np.float64) / scale / source
                dy = table["motion_y"].astype(np.float64) / scale / source
                past = source < 0
                dx[past] /= past_gap
                dy[past] /= past_gap
                area = (table["w"].astype(np.float64) * table["h"])
                if threshold > 0:
                    keep = np.hypot(dx, dy) >= threshold
                    dx, dy, area = dx * keep, dy * keep, area * keep
                #: One cell per vector, by integer division, with no loop and no
                #: painting across cells.
                #:
                #: That is exact rather than approximate, because of how these codecs
                #: partition: every H.264 partition --- 16x16 down to 4x4 --- lies
                #: INSIDE a single macroblock, and MPEG-4 Part 2 uses 16x16 and 8x8. So
                #: a block never straddles a 16-pixel cell boundary, checked on a real
                #: encode: 0 of 12,130 vectors crossed one. `dst` is the block centre,
                #: so `dst // 16` is its macroblock.
                #:
                #: The loop this replaces ran once per vector. At 1920x1080 that is
                #: 8,160 macroblocks a frame, and 2.5 billion iterations over a
                #: 100-minute recording --- it worked on a 60-frame test clip and would
                #: never have finished on a session.
                r = np.clip(table["dst_y"].astype(np.int64) // _GRID, 0, rows - 1)
                c = np.clip(table["dst_x"].astype(np.int64) // _GRID, 0, cols - 1)
                flat = r * cols + c
                size = rows * cols
                w += np.bincount(flat, weights=area, minlength=size).reshape(rows, cols)
                vx += np.bincount(flat, weights=dx * area,
                                  minlength=size).reshape(rows, cols)
                vy += np.bincount(flat, weights=dy * area,
                                  minlength=size).reshape(rows, cols)
            busy = w > 0
            vx[busy] /= w[busy]
            vy[busy] /= w[busy]
            yield vx, vy, w * np.hypot(vx, vy), t, is_p
    finally:
        container.close()

accumulate_motion_vectors

accumulate_motion_vectors(filename, p_frames_only=True, deterministic=False, threshold=0.0)

The whole space motion happened in, summed over the recording.

Returns (weight, vx, vy, coherence): how much motion each cell saw, the direction it saw on average, and how consistently that direction held.

P-frames only by default. A B-frame's vectors are referenced over varying temporal distances, and pooling them into a spatial average blurs the direction that is the reason for accumulating vectors rather than frame differences.

Source code in musicalgestures/_motionvectors.py
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
def accumulate_motion_vectors(filename, p_frames_only=True, deterministic=False, threshold=0.0):
    """The whole space motion happened in, summed over the recording.

    Returns `(weight, vx, vy, coherence)`: how much motion each cell saw, the
    direction it saw on average, and how consistently that direction held.

    P-frames only by default. A B-frame's vectors are referenced over varying temporal
    distances, and pooling them into a spatial average blurs the direction that is the
    reason for accumulating vectors rather than frame differences.
    """
    import numpy as np

    total = sum_x = sum_y = sum_speed = None
    for vx, vy, weight, _, is_p in motion_vector_grid(filename, deterministic, threshold):
        if p_frames_only and not is_p:
            continue
        if total is None:
            total = np.zeros_like(weight)
            sum_x = np.zeros_like(weight)
            sum_y = np.zeros_like(weight)
            sum_speed = np.zeros_like(weight)
        total += weight
        sum_x += vx * weight
        sum_y += vy * weight
        sum_speed += np.hypot(vx, vy) * weight
    if total is None:
        z = np.zeros((1, 1))
        return z, z.copy(), z.copy(), z.copy()
    busy = total > 0
    mean_x = np.zeros_like(total)
    mean_y = np.zeros_like(total)
    coherence = np.zeros_like(total)
    mean_x[busy] = sum_x[busy] / total[busy]
    mean_y[busy] = sum_y[busy] / total[busy]
    #: How consistently this cell moved ONE way. The length of the mean vector over the
    #: mean length: 1 where every crossing went the same direction, near 0 where they
    #: cancelled. Without it a cell that saw a dancer pass back and forth all afternoon
    #: is indistinguishable from one that saw nothing but encoder noise, since both have
    #: a near-zero mean vector and an arbitrary direction.
    #:
    #: LIMIT. P-frames arrive at roughly a quarter of the frame rate, so a motion that
    #: reverses faster than a few samples per cycle is under-sampled and reads as
    #: coherent when it is not: measured on a block oscillating at 3 samples per cycle
    #: this returns 0.97, and at 12 samples per cycle it returns 0.06. On 50 fps footage
    #: that puts the honest ceiling around 1.5 Hz --- fine for a dancer crossing a room,
    #: wrong for a shaking hand.
    mean_speed = np.zeros_like(total)
    mean_speed[busy] = sum_speed[busy] / total[busy]
    moving = mean_speed > 0
    coherence[moving] = (np.hypot(mean_x, mean_y)[moving] / mean_speed[moving])
    return total, mean_x, mean_y, np.clip(coherence, 0, 1)

mg_motionvectorhistory

mg_motionvectorhistory(self, mode='direction', colormap='inferno', gamma=0.5, target_name=None, overwrite=True)

The whole space the motion happened in, accumulated from the codec's vectors.

motionhistory() renders the Bobick--Davis image from frame differences and encodes recency; heatmap() accumulates where pixels changed. Neither can say which way anything went, because a frame difference has no sign --- somebody entering a region and leaving it look identical. A motion vector is a displacement, so this can, and that is the reason for it to exist.

Parameters:

Name Type Description Default
mode str

'direction' colours each cell by the direction motion took there, hue running around the compass, with brightness for how much motion the cell saw. 'magnitude' drops the direction and applies colormap to the amount alone, which is the comparable view when direction is a distraction. Defaults to 'direction'.

'direction'
colormap str

Matplotlib colormap, used by 'magnitude'. Defaults to 'inferno'.

'inferno'
gamma float

Applied to the accumulated amount before colouring, so that a few violent moments do not leave everything else black. Defaults to 0.5.

0.5
target_name str

Output path. Defaults to the input name with _motionvectorhistory.

None
overwrite bool

Defaults to True.

True

Returns:

Name Type Description
MgImage MgImage

the rendered image, at the video's own dimensions.

Notes

Built from P-frames only, for the reason given on motionvectordata. And the vectors are the encoder's decisions: over a region with no detail every candidate predicts equally well, so the encoder copies whatever its neighbours used and the field there is inherited rather than observed. Expect that wherever the picture is flat.

Source code in musicalgestures/_motionvectors.py
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
def mg_motionvectorhistory(self: "musicalgestures.MgVideo", mode: str = "direction",
                           colormap: str = "inferno", gamma: float = 0.5,
                           target_name: str | None = None,
                           overwrite: bool = True) -> "MgImage":
    """The whole space the motion happened in, accumulated from the codec's vectors.

    `motionhistory()` renders the Bobick--Davis image from frame differences and encodes
    *recency*; `heatmap()` accumulates where pixels changed. Neither can say which way
    anything went, because a frame difference has no sign --- somebody entering a region
    and leaving it look identical. A motion vector is a displacement, so this can, and
    that is the reason for it to exist.

    Args:
        mode (str, optional): `'direction'` colours each cell by the direction motion
            took there, hue running around the compass, with brightness for how much
            motion the cell saw. `'magnitude'` drops the direction and applies `colormap`
            to the amount alone, which is the comparable view when direction is a
            distraction. Defaults to `'direction'`.
        colormap (str, optional): Matplotlib colormap, used by `'magnitude'`. Defaults to
            `'inferno'`.
        gamma (float, optional): Applied to the accumulated amount before colouring, so
            that a few violent moments do not leave everything else black. Defaults to
            0.5.
        target_name (str, optional): Output path. Defaults to the input name with
            `_motionvectorhistory`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgImage: the rendered image, at the video's own dimensions.

    Notes:
        Built from P-frames only, for the reason given on `motionvectordata`. And the
        vectors are the encoder's decisions: over a region with no detail every candidate
        predicts equally well, so the encoder copies whatever its neighbours used and the
        field there is inherited rather than observed. Expect that wherever the picture is
        flat.
    """
    import cv2
    import matplotlib
    import numpy as np

    from musicalgestures._utils import MgImage

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_motionvectorhistory.png', target_name, overwrite)

    weight, vx, vy, coherence = accumulate_motion_vectors(self.filename)
    #: Normalised against a high percentile, not the maximum. Over a long recording the
    #: distribution is heavy-tailed and dividing by the maximum leaves the typical cell
    #: close to it, so the whole frame comes out uniformly bright and shows nothing.
    ceiling = np.percentile(weight[weight > 0], 99) if (weight > 0).any() else 1.0
    amount = np.clip(weight / max(ceiling, 1e-9), 0, 1)
    if gamma and gamma != 1.0:
        amount = np.power(amount, gamma)

    if mode == "direction":
        #: Hue is the compass bearing, so opposite directions are opposite colours.
        #: Value is how much motion there was, and **saturation is how consistently it
        #: went that way**. Pinning saturation at full was the first version's mistake:
        #: it painted every cell a vivid colour whether or not the direction meant
        #: anything, and since every cell accumulates something over a hundred minutes,
        #: the result was a uniform wash. Grey now reads as "moved, but not any one
        #: way", which is the honest thing to say about most of a room.
        hue = (np.arctan2(vy, vx) + np.pi) / (2 * np.pi) * 179
        hsv = np.stack([hue.astype(np.uint8),
                        np.clip(coherence * 255, 0, 255).astype(np.uint8),
                        np.clip(amount * 255, 0, 255).astype(np.uint8)], axis=-1)
        rgb = cv2.cvtColor(hsv, cv2.COLOR_HSV2RGB)
    elif mode == "magnitude":
        cmap = matplotlib.colormaps[colormap]
        rgb = (cmap(amount)[..., :3] * 255).astype(np.uint8)
    elif mode == "coherence":
        cmap = matplotlib.colormaps[colormap]
        rgb = (cmap(coherence)[..., :3] * 255).astype(np.uint8)
    else:
        raise ValueError(
            f"mode must be 'direction', 'magnitude' or 'coherence', not {mode!r}")

    width, height = self.width, self.height
    full = cv2.resize(rgb, (width, height), interpolation=cv2.INTER_NEAREST)
    cv2.imwrite(target_name, full[..., ::-1])

    self.motionvectorhistory_image = MgImage(target_name)
    return self.motionvectorhistory_image

motion_vector_motiongrams

motion_vector_motiongrams(filename, p_frames_only=True, deterministic=False, threshold=0.0)

The two classic motiongrams from the vectors, named for the picture.

The HORIZONTAL motiongram is the wide picture: each frame's grid collapsed across its columns to a y-profile, stacked so time runs rightward and vertical position down. The VERTICAL motiongram is the tall picture: each frame collapsed down its rows to an x-profile, stacked so time runs downward and horizontal position across. A body crossing the room draws a diagonal in the vertical gram; a body rising or falling draws one in the horizontal. The orientations and names are the ones motiongrams() has always used.

Returns:

Name Type Description
tuple

(horizontal, vertical), shaped (rows, frames) and (frames, cols).

Source code in musicalgestures/_motionvectors.py
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
def motion_vector_motiongrams(filename, p_frames_only=True, deterministic=False, threshold=0.0):
    """The two classic motiongrams from the vectors, named for the picture.

    The HORIZONTAL motiongram is the wide picture: each frame's grid collapsed across
    its columns to a y-profile, stacked so time runs rightward and vertical position
    down. The VERTICAL motiongram is the tall picture: each frame collapsed down its
    rows to an x-profile, stacked so time runs downward and horizontal position across.
    A body crossing the room draws a diagonal in the vertical gram; a body rising or
    falling draws one in the horizontal. The orientations and names are the ones
    `motiongrams()` has always used.

    Returns:
        tuple: `(horizontal, vertical)`, shaped (rows, frames) and (frames, cols).
    """
    import numpy as np

    #: Only the REDUCED column is kept per frame, never the grid it came from. A
    #: motiongram must grow with the recording; the full field behind it must not.
    across, down = [], []
    for _, _, weight, _, is_p in motion_vector_grid(filename, deterministic, threshold):
        if p_frames_only and not is_p:
            continue
        across.append(weight.sum(axis=0))
        down.append(weight.sum(axis=1))
    if not across:
        return np.zeros((0, 0)), np.zeros((0, 0))
    #: The horizontal gram transposed so time runs along the image's x axis; the
    #: vertical one left with time down its rows. Each is then already the picture.
    return np.array(down).T, np.array(across)

mg_motionvectorgrams

mg_motionvectorgrams(self, colormap='inferno', gamma=0.5, max_width=4000, target_name=None, overwrite=True)

Motiongrams built from the codec's motion vectors.

The same view motiongrams() gives, at a fraction of the cost, because the displacement has already been computed by the encoder. The trade is resolution: this is drawn on the macroblock lattice, sixteen pixels, and on P-frames only, so on 50 fps footage it carries about 12.6 columns per second rather than 50.

Parameters:

Name Type Description Default
colormap str

Matplotlib colormap. Defaults to 'inferno'.

'inferno'
gamma float

Applied before colouring so quiet passages stay visible. Defaults to 0.5.

0.5
max_width int

Longest the saved image's TIME axis may be, in pixels --- across the horizontal image, down the vertical one. One point per P-frame is right for the array and wrong for the picture: a 158-minute session has 77,592 of them, and stretching that to the video's height gave a 149-megapixel PNG of 309 MB for something meant to be glanced at. Points are pooled by taking the maximum, not the mean, so a brief accent still shows in a column standing for several seconds. Defaults to 4000.

4000
target_name str

Output path; the two images take _mvgram_y and _mvgram_x from it, named by the position axis each keeps. Defaults to the input name.

None
overwrite bool

Defaults to True.

True

Returns:

Name Type Description
MgList MgList

the y-motiongram and the x-motiongram, in that order, in the classic

MgList

orientations --- y wide with time rightward, x tall with time downward.

Source code in musicalgestures/_motionvectors.py
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
def mg_motionvectorgrams(self: "musicalgestures.MgVideo", colormap: str = "inferno",
                         gamma: float = 0.5, max_width: int = 4000,
                         target_name: str | None = None,
                         overwrite: bool = True) -> "MgList":
    """Motiongrams built from the codec's motion vectors.

    The same view `motiongrams()` gives, at a fraction of the cost, because the
    displacement has already been computed by the encoder. The trade is resolution: this
    is drawn on the macroblock lattice, sixteen pixels, and on P-frames only, so on 50 fps
    footage it carries about 12.6 columns per second rather than 50.

    Args:
        colormap (str, optional): Matplotlib colormap. Defaults to `'inferno'`.
        gamma (float, optional): Applied before colouring so quiet passages stay visible.
            Defaults to 0.5.
        max_width (int, optional): Longest the saved image's TIME axis may be, in
            pixels --- across the horizontal image, down the vertical one. One point per
            P-frame is right for the array and wrong for the picture: a 158-minute
            session has 77,592 of them, and stretching that to the video's height gave a
            149-megapixel PNG of 309 MB for something meant to be glanced at. Points are
            pooled by taking the maximum, not the mean, so a brief accent still shows in
            a column standing for several seconds. Defaults to 4000.
        target_name (str, optional): Output path; the two images take `_mvgram_y` and
            `_mvgram_x` from it, named by the position axis each keeps. Defaults to
            the input name.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgList: the y-motiongram and the x-motiongram, in that order, in the classic
        orientations --- y wide with time rightward, x tall with time downward.
    """
    import cv2
    import matplotlib
    import numpy as np

    from musicalgestures._mglist import MgList
    from musicalgestures._utils import MgImage

    of, fex = os.path.splitext(self.filename)
    base = resolve_filename(of, '_mvgram.png', target_name, overwrite)
    stem, _ = os.path.splitext(base)

    horizontal, vertical = motion_vector_motiongrams(self.filename)
    cmap = matplotlib.colormaps[colormap]
    images = []
    #: Classic orientations, named for the picture: the horizontal gram is wide with
    #: time rightward, the vertical gram is TALL with time downward. The time axis is
    #: axis 1 of the first and axis 0 of the second, so everything below works on
    #: (space, time) and transposes the vertical picture back at the end.
    for gram, suffix, span, tall in ((horizontal, "_y", self.height, False),
                                     (vertical.T, "_x", self.width, True)):
        if gram.size and gram.shape[1] > max_width:
            #: Pool by maximum rather than mean. A column here stands for several seconds
            #: once a long recording is squeezed to a readable length, and averaging
            #: would bury a one-second accent under the quiet around it --- which is
            #: exactly what a motiongram is read for.
            edges = np.linspace(0, gram.shape[1], max_width + 1).astype(int)
            gram = np.stack([gram[:, a:max(b, a + 1)].max(axis=1)
                             for a, b in zip(edges[:-1], edges[1:])], axis=1)
        scaled = gram / gram.max() if gram.size and gram.max() > 0 else gram
        if gamma and gamma != 1.0:
            scaled = np.power(scaled, gamma)
        rgb = (cmap(scaled)[..., :3] * 255).astype(np.uint8)
        #: Stretched back to the frame's own dimension on the spatial axis so the two
        #: grams line up with the video and with each other; nearest, because smoothing a
        #: macroblock lattice invents detail the vectors never had.
        out = cv2.resize(rgb, (rgb.shape[1], span), interpolation=cv2.INTER_NEAREST)
        if tall:
            out = out.transpose(1, 0, 2)
        path = f"{stem}{suffix}.png"
        cv2.imwrite(path, out[..., ::-1])
        images.append(MgImage(path))

    self.motionvectorgrams_images = MgList(images)
    return self.motionvectorgrams_images

motion_vector_profiles

motion_vector_profiles(filename, n_samples=40, axis='horizontal', p_frames_only=True, deterministic=False, threshold=0.0)

Motion profiles at n_samples moments, for stacking as a waterfall.

Each profile is one spatial axis of the vector grid, summed over the other, pooled over the frames nearest that moment rather than taken from a single frame --- a single P-frame's vectors are sparse, and a waterfall of sparse profiles reads as noise where the pooled version reads as a body moving.

Returns:

Name Type Description
tuple

(profiles, times), profiles shaped (n_samples, cells).

Source code in musicalgestures/_motionvectors.py
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
def motion_vector_profiles(filename, n_samples=40, axis="horizontal",
                           p_frames_only=True, deterministic=False, threshold=0.0):
    """Motion profiles at `n_samples` moments, for stacking as a waterfall.

    Each profile is one spatial axis of the vector grid, summed over the other, pooled
    over the frames nearest that moment rather than taken from a single frame --- a
    single P-frame's vectors are sparse, and a waterfall of sparse profiles reads as
    noise where the pooled version reads as a body moving.

    Returns:
        tuple: `(profiles, times)`, profiles shaped (n_samples, cells).
    """
    import numpy as np

    per_frame, times = [], []
    for _, _, weight, t, is_p in motion_vector_grid(filename, deterministic, threshold):
        if p_frames_only and not is_p:
            continue
        per_frame.append(weight.sum(axis=0) if axis == "horizontal"
                         else weight.sum(axis=1))
        times.append(t)
    if not per_frame:
        return np.zeros((0, 0)), np.zeros(0)
    per_frame = np.array(per_frame)
    times = np.array(times)
    edges = np.linspace(0, len(per_frame), n_samples + 1).astype(int)
    profiles, moments = [], []
    for a, b in zip(edges[:-1], edges[1:]):
        b = max(b, a + 1)
        profiles.append(per_frame[a:b].sum(axis=0))
        moments.append(float(times[a:b].mean()))
    return np.array(profiles), np.array(moments)

mg_motionvectorwaterfall

mg_motionvectorwaterfall(self, n_samples=40, axis='horizontal', cmap='viridis', dpi=200, elev=35, azim=-60, target_name=None, overwrite=True)

A waterfall of motion profiles, cascading through time.

The same cascade silhouette_waterfall() draws, but from displacement rather than from a silhouette, so it needs no background subtraction and no pose model --- and it shows where motion was, not where a body was. A dancer standing still has a silhouette and no vectors.

Parameters:

Name Type Description Default
n_samples int

How many profiles to stack. Defaults to 40.

40
axis str

'horizontal' profiles over x, 'vertical' over y. Defaults to 'horizontal'.

'horizontal'
cmap str

Matplotlib colormap, applied over time. Defaults to 'viridis'.

'viridis'
dpi int

Defaults to 200.

200
elev, azim float

3D view angles. Default to 35 and -60.

required
target_name str

Output path. Defaults to the input name with _motionvectorwaterfall.

None
overwrite bool

Defaults to True.

True

Returns:

Name Type Description
MgFigure MgFigure

the rendered waterfall.

Source code in musicalgestures/_motionvectors.py
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
def mg_motionvectorwaterfall(self: "musicalgestures.MgVideo", n_samples: int = 40,
                             axis: str = "horizontal", cmap: str = "viridis",
                             dpi: int = 200, elev: float = 35, azim: float = -60,
                             target_name: str | None = None,
                             overwrite: bool = True) -> "MgFigure":
    """A waterfall of motion profiles, cascading through time.

    The same cascade `silhouette_waterfall()` draws, but from displacement rather than
    from a silhouette, so it needs no background subtraction and no pose model --- and it
    shows where motion was, not where a body was. A dancer standing still has a
    silhouette and no vectors.

    Args:
        n_samples (int, optional): How many profiles to stack. Defaults to 40.
        axis (str, optional): `'horizontal'` profiles over x, `'vertical'` over y.
            Defaults to `'horizontal'`.
        cmap (str, optional): Matplotlib colormap, applied over time. Defaults to
            `'viridis'`.
        dpi (int, optional): Defaults to 200.
        elev, azim (float, optional): 3D view angles. Default to 35 and -60.
        target_name (str, optional): Output path. Defaults to the input name with
            `_motionvectorwaterfall`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgFigure: the rendered waterfall.
    """
    import matplotlib
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._utils import MgFigure

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_motionvectorwaterfall.png', target_name,
                                   overwrite)

    profiles, times = motion_vector_profiles(self.filename, n_samples=n_samples,
                                             axis=axis)
    fig = plt.figure(figsize=(10, 6), dpi=dpi)
    ax = fig.add_subplot(111, projection="3d")
    colours = matplotlib.colormaps[cmap](np.linspace(0, 1, max(len(profiles), 1)))
    x = np.arange(profiles.shape[1]) if profiles.size else np.arange(0)
    for i, profile in enumerate(profiles):
        ax.plot(x, np.full_like(x, times[i], dtype=float), profile,
                color=colours[i], linewidth=0.9)
    ax.set_xlabel("x" if axis == "horizontal" else "y")
    ax.set_ylabel("time (s)")
    ax.set_zlabel("motion")
    ax.view_init(elev=elev, azim=azim)
    fig.tight_layout()
    fig.savefig(target_name)
    plt.close(fig)

    self.motionvectorwaterfall_figure = MgFigure(
        figure=None, figure_type="video.motionvectorwaterfall",
        data={"profiles": profiles, "times": times}, layers=None,
        image=target_name)
    return self.motionvectorwaterfall_figure

motion_vector_views

motion_vector_views(filename, p_frames_only=True, deterministic=False, threshold=0.0)

Every view at once, in one decode.

The individual functions each decode the file, which is nearly all of the cost, so asking for four views costs four times what it needs to. This walks the vectors once and accumulates all of them. On a 100-minute 1920x1080 recording that is about four minutes rather than twenty-four.

The result is checked against the individual functions in the test suite rather than assumed equal to them, because a fast path that quietly disagrees with the slow one is worse than no fast path.

Source code in musicalgestures/_motionvectors.py
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
def motion_vector_views(filename, p_frames_only=True, deterministic=False, threshold=0.0):
    """Every view at once, in one decode.

    The individual functions each decode the file, which is nearly all of the cost, so
    asking for four views costs four times what it needs to. This walks the vectors once
    and accumulates all of them. On a 100-minute 1920x1080 recording that is about four
    minutes rather than twenty-four.

    The result is checked against the individual functions in the test suite rather than
    assumed equal to them, because a fast path that quietly disagrees with the slow one
    is worse than no fast path.
    """
    import numpy as np

    total = sum_x = sum_y = sum_speed = None
    times, magnitude, across, down = [], [], [], []
    for vx, vy, weight, t, is_p in motion_vector_grid(filename, deterministic, threshold):
        if p_frames_only and not is_p:
            continue
        if total is None:
            total = np.zeros_like(weight)
            sum_x = np.zeros_like(weight)
            sum_y = np.zeros_like(weight)
            sum_speed = np.zeros_like(weight)
        total += weight
        sum_x += vx * weight
        sum_y += vy * weight
        sum_speed += np.hypot(vx, vy) * weight
        times.append(t)
        magnitude.append(float(weight.sum()))
        across.append(weight.sum(axis=0))
        down.append(weight.sum(axis=1))

    if total is None:
        z = np.zeros((1, 1))
        return MgMotionVectorViews(z, z.copy(), z.copy(), z.copy(), np.zeros(0),
                                   np.zeros(0), np.zeros((0, 0)), np.zeros((0, 0)))

    busy = total > 0
    mean_x = np.zeros_like(total)
    mean_y = np.zeros_like(total)
    mean_speed = np.zeros_like(total)
    coherence = np.zeros_like(total)
    mean_x[busy] = sum_x[busy] / total[busy]
    mean_y[busy] = sum_y[busy] / total[busy]
    mean_speed[busy] = sum_speed[busy] / total[busy]
    moving = mean_speed > 0
    coherence[moving] = np.hypot(mean_x, mean_y)[moving] / mean_speed[moving]
    return MgMotionVectorViews(
        history_weight=total, history_vx=mean_x, history_vy=mean_y,
        history_coherence=np.clip(coherence, 0, 1),
        time=np.array(times), magnitude=np.array(magnitude),
        motiongram_y=np.array(down).T,
        motiongram_x=np.array(across))

mg_motionvectoroverview

mg_motionvectoroverview(self, colormap='inferno', gamma=0.5, target_name=None, overwrite=True)

One sheet per recording, from one decode: where, how much, and when.

Built for a first look at unfamiliar material rather than for measurement. Hours of archive video, several representations per file, as the starting point for qualitative annotation --- so it favours covering the whole recording cheaply over resolving any part of it finely.

Five panels, of the three kinds a first pass wants:

  • spatial --- the area motion covered, coloured by direction, and the same accumulation as plain amount;
  • temporal --- motion over the whole recording, the curve to scrub against;
  • spatio-temporal --- the y- and x-motiongrams, position against time, where a body crossing the room draws a diagonal.

Everything comes from the codec's own motion vectors, so the cost is close to the cost of reading the file: about five minutes for a 100-minute 1920x1080 recording, against roughly twenty-four for the same views computed one at a time, and far less than differencing the pixels.

Parameters:

Name Type Description Default
colormap str

Matplotlib colormap for the amount panels. Defaults to 'inferno'.

'inferno'
gamma float

Applied before colouring so quiet passages stay visible. Defaults to 0.5.

0.5
target_name str

Output path. Defaults to the input name with _motionvectoroverview.

None
overwrite bool

Defaults to True.

True

Returns:

Name Type Description
MgFigure MgFigure

the sheet, with every underlying array in .data so the same pass can

MgFigure

feed further analysis without decoding again.

Source code in musicalgestures/_motionvectors.py
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
def mg_motionvectoroverview(self: "musicalgestures.MgVideo", colormap: str = "inferno",
                            gamma: float = 0.5, target_name: str | None = None,
                            overwrite: bool = True) -> "MgFigure":
    """One sheet per recording, from one decode: where, how much, and when.

    Built for a first look at unfamiliar material rather than for measurement. Hours of
    archive video, several representations per file, as the starting point for
    qualitative annotation --- so it favours covering the whole recording cheaply over
    resolving any part of it finely.

    Five panels, of the three kinds a first pass wants:

    * **spatial** --- the area motion covered, coloured by direction, and the same
      accumulation as plain amount;
    * **temporal** --- motion over the whole recording, the curve to scrub against;
    * **spatio-temporal** --- the y- and x-motiongrams, position against time,
      where a body crossing the room draws a diagonal.

    Everything comes from the codec's own motion vectors, so the cost is close to the
    cost of reading the file: about five minutes for a 100-minute 1920x1080 recording,
    against roughly twenty-four for the same views computed one at a time, and far less
    than differencing the pixels.

    Args:
        colormap (str, optional): Matplotlib colormap for the amount panels. Defaults to
            `'inferno'`.
        gamma (float, optional): Applied before colouring so quiet passages stay visible.
            Defaults to 0.5.
        target_name (str, optional): Output path. Defaults to the input name with
            `_motionvectoroverview`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgFigure: the sheet, with every underlying array in `.data` so the same pass can
        feed further analysis without decoding again.
    """
    import cv2
    import matplotlib
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._utils import MgFigure

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_motionvectoroverview.png', target_name,
                                   overwrite)
    v = motion_vector_views(self.filename)

    def shade(a):
        ceiling = np.percentile(a[a > 0], 99) if (a > 0).any() else 1.0
        return np.power(np.clip(a / max(ceiling, 1e-9), 0, 1), gamma)

    hue = (np.arctan2(v.history_vy, v.history_vx) + np.pi) / (2 * np.pi) * 179
    direction = cv2.cvtColor(np.stack([
        hue.astype(np.uint8),
        np.clip(v.history_coherence * 255, 0, 255).astype(np.uint8),
        np.clip(shade(v.history_weight) * 255, 0, 255).astype(np.uint8)], axis=-1),
        cv2.COLOR_HSV2RGB)

    cmap = matplotlib.colormaps[colormap]
    fig = plt.figure(figsize=(16, 11), dpi=110)
    grid = fig.add_gridspec(3, 2, height_ratios=[3, 1, 2], hspace=0.28, wspace=0.12)

    ax = fig.add_subplot(grid[0, 0])
    ax.imshow(direction, aspect="auto", interpolation="nearest")
    ax.set_title("where, and which way (hue = direction, saturation = consistency)")
    ax.set_xticks([]); ax.set_yticks([])

    ax = fig.add_subplot(grid[0, 1])
    ax.imshow(shade(v.history_weight), cmap=colormap, aspect="auto",
              interpolation="nearest")
    ax.set_title("where, as amount")
    ax.set_xticks([]); ax.set_yticks([])

    ax = fig.add_subplot(grid[1, :])
    if len(v.time):
        ax.plot(v.time / 60, v.magnitude, linewidth=0.4, color="#c1440e")
        ax.set_xlim(v.time[0] / 60, v.time[-1] / 60)
    ax.set_title("how much, over the recording")
    ax.set_xlabel("minutes")
    ax.set_yticks([])

    minutes = (v.time[-1] / 60) if len(v.time) else 1
    #: Classic orientations, named for the picture: the horizontal gram wide with time
    #: rightward, the vertical gram with time running DOWNWARD in its panel.
    ax = fig.add_subplot(grid[2, 0])
    if v.motiongram_y.size:
        ax.imshow(shade(v.motiongram_y), cmap=colormap, aspect="auto",
                  interpolation="nearest",
                  extent=(0, minutes, v.motiongram_y.shape[0], 0))
    ax.set_title("y-motiongram (y down, time \u2192)")
    ax.set_xlabel("minutes")
    ax.set_yticks([])

    ax = fig.add_subplot(grid[2, 1])
    if v.motiongram_x.size:
        ax.imshow(shade(v.motiongram_x), cmap=colormap, aspect="auto",
                  interpolation="nearest",
                  extent=(0, v.motiongram_x.shape[1], minutes, 0))
    ax.set_title("x-motiongram (x across, time \u2193)")
    ax.set_ylabel("minutes")
    ax.set_xticks([])

    fig.suptitle(os.path.basename(self.filename), fontsize=11)
    fig.savefig(target_name, bbox_inches="tight")
    plt.close(fig)

    self.motionvectoroverview_figure = MgFigure(
        figure=None, figure_type="video.motionvectoroverview",
        data={"time": v.time, "magnitude": v.magnitude,
              "history_weight": v.history_weight, "history_vx": v.history_vx,
              "history_vy": v.history_vy, "history_coherence": v.history_coherence,
              "motiongram_y": v.motiongram_y,
              "motiongram_x": v.motiongram_x},
        layers=None, image=target_name)
    return self.motionvectoroverview_figure

motion_scape

motion_scape(track, n_scales=64)

A Sapp-style scape of any per-frame track: every timescale at once.

In Craig Sapp's keyscape each row is a window length and each cell the key that best fits that window, so the plot answers "at what scale does this look like one thing" rather than "what is happening now". The same construction over quantity of motion reads the same way: a recording that is one long even stretch looks flat at every scale, and a few busy patches separated by quiet stay separate at the bottom and merge into one bright mass as the windows grow.

Rows run coarse to fine --- the top row is a single window covering the whole recording, the bottom row is the track itself --- which puts the apex at the top, as a keyscape has it.

Computed from a cumulative sum, so the cost does not depend on how wide the windows get: the whole plot is O(n_scales x n) whatever the recording's length.

Parameters:

Name Type Description Default
track ndarray

One value per frame. Quantity of motion from either source works; nothing here assumes it came from motion vectors.

required
n_scales int

How many window lengths to stack. Defaults to 64.

64

Returns:

Type Description

np.ndarray: shape (n_scales, len(track)), coarsest row first. Each cell is the

mean of the track over a window of that row's length centred on that position,

and NaN where a window that long does not fit --- which is what gives the plot

its triangular shape, apex at the top.

Source code in musicalgestures/_motionvectors.py
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
def motion_scape(track, n_scales=64):
    """A Sapp-style scape of any per-frame track: every timescale at once.

    In Craig Sapp's keyscape each row is a window length and each cell the key that best
    fits that window, so the plot answers "at what scale does this look like one thing"
    rather than "what is happening now". The same construction over quantity of motion
    reads the same way: a recording that is one long even stretch looks flat at every
    scale, and a few busy patches separated by quiet stay separate at the bottom and merge
    into one bright mass as the windows grow.

    Rows run coarse to fine --- the top row is a single window covering the whole
    recording, the bottom row is the track itself --- which puts the apex at the top, as
    a keyscape has it.

    Computed from a cumulative sum, so the cost does not depend on how wide the windows
    get: the whole plot is O(n_scales x n) whatever the recording's length.

    Args:
        track (np.ndarray): One value per frame. Quantity of motion from either source
            works; nothing here assumes it came from motion vectors.
        n_scales (int, optional): How many window lengths to stack. Defaults to 64.

    Returns:
        np.ndarray: shape (n_scales, len(track)), coarsest row first. Each cell is the
        mean of the track over a window of that row's length centred on that position,
        and NaN where a window that long does not fit --- which is what gives the plot
        its triangular shape, apex at the top.
    """
    import numpy as np

    q = np.asarray(track, dtype=np.float64).ravel()
    n = len(q)
    if n == 0:
        return np.zeros((n_scales, 0))
    #: One prepended zero so that `cumulative[b] - cumulative[a]` is the sum over [a, b).
    cumulative = np.concatenate([[0.0], np.cumsum(q)])
    positions = np.arange(n)
    #: Geometric, because the interesting structure is spread over orders of magnitude:
    #: linear scales would spend most of the plot on windows that all look alike.
    #: Geometric spacing rounds to duplicate integers at the short end --- for 20 scales
    #: over 500 frames it yields 18 distinct lengths --- so the shortfall is topped up
    #: from the unused lengths rather than by nudging the largest, which was an infinite
    #: loop: the nudged value already existed, so the set never grew.
    chosen = sorted({int(x) for x in np.geomspace(1, n, n_scales)})
    if len(chosen) < n_scales:
        spare = [x for x in range(1, n + 1) if x not in set(chosen)]
        chosen = sorted(chosen + spare[:n_scales - len(chosen)])
    while len(chosen) < n_scales:                 # fewer frames than scales asked for
        chosen.append(chosen[-1])
    lengths = np.array(chosen[:n_scales])

    scape = np.full((len(lengths), n), np.nan, dtype=np.float64)
    for row, length in enumerate(sorted(lengths, reverse=True)):
        half = length // 2
        lo = positions - half
        hi = lo + length
        #: Only where the whole window fits. Truncating at the edges instead would fill
        #: the corners with partial windows and square the plot off --- and the triangle
        #: IS the information: a row is only as wide as the number of places a window
        #: that long can sit, so the apex is the single window covering everything.
        fits = (lo >= 0) & (hi <= n)
        scape[row, fits] = ((cumulative[hi[fits]] - cumulative[lo[fits]])
                            / max(length, 1))
    return scape

mg_motionscape

mg_motionscape(self, track=None, track_times=None, n_scales=96, colormap='magma', gamma=0.5, target_name=None, overwrite=True)

A scape of the recording's motion: every timescale in one triangle.

After Craig Sapp's keyscape, where each row is a window length and the apex is a single window covering the whole piece. Here each cell is the mean quantity of motion over that window, so the plot answers a question the flat curve cannot: at what scale does this recording stop looking like one thing? A session of even, continuous improvisation is flat all the way up. A session of separated bursts stays separated at the base and merges into one mass near the apex, and the height at which it merges is the length of the structure.

Parameters:

Name Type Description Default
track array - like

One value per frame to build the scape from. Pass a frame-differenced quantity of motion here --- mg_motion's QomRaw, or a qom.f4 read off disk --- to scape that instead. Nothing about the construction assumes where the numbers came from. Defaults to None, which reads the codec's motion vectors, cheap enough to run over an archive.

None
track_times array - like

Seconds for each value in track. Needed whenever the track is not one value per video frame --- notably motionvectoroverview's own magnitude, which is P-frames only and so arrives at about a quarter of the frame rate. Without it the time axis is wrong by that factor while looking perfectly reasonable. Defaults to None, meaning the track is at the video's frame rate.

None
n_scales int

Rows in the triangle. Defaults to 96.

96
colormap str

Defaults to 'magma'.

'magma'
gamma float

Applied before colouring. Defaults to 0.5.

0.5
target_name str

Output path. Defaults to the input name with _motionscape.

None
overwrite bool

Defaults to True.

True

Returns:

Name Type Description
MgFigure MgFigure

the scape, with the track and the triangle in .data.

Source code in musicalgestures/_motionvectors.py
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
def mg_motionscape(self: "musicalgestures.MgVideo", track=None, track_times=None,
                   n_scales: int = 96, colormap: str = "magma",
                   gamma: float = 0.5, target_name: str | None = None,
                   overwrite: bool = True) -> "MgFigure":
    """A scape of the recording's motion: every timescale in one triangle.

    After Craig Sapp's keyscape, where each row is a window length and the apex is a
    single window covering the whole piece. Here each cell is the mean quantity of motion
    over that window, so the plot answers a question the flat curve cannot: **at what
    scale does this recording stop looking like one thing?** A session of even, continuous
    improvisation is flat all the way up. A session of separated bursts stays separated at
    the base and merges into one mass near the apex, and the height at which it merges is
    the length of the structure.

    Args:
        track (array-like, optional): One value per frame to build the scape from. Pass
            a frame-differenced quantity of motion here --- `mg_motion`'s `QomRaw`, or a
            `qom.f4` read off disk --- to scape that instead. Nothing about the
            construction assumes where the numbers came from. Defaults to None, which
            reads the codec's motion vectors, cheap enough to run over an archive.
        track_times (array-like, optional): Seconds for each value in `track`. Needed
            whenever the track is not one value per video frame --- notably
            `motionvectoroverview`'s own `magnitude`, which is P-frames only and so
            arrives at about a quarter of the frame rate. Without it the time axis is
            wrong by that factor while looking perfectly reasonable. Defaults to None,
            meaning the track is at the video's frame rate.
        n_scales (int, optional): Rows in the triangle. Defaults to 96.
        colormap (str, optional): Defaults to `'magma'`.
        gamma (float, optional): Applied before colouring. Defaults to 0.5.
        target_name (str, optional): Output path. Defaults to the input name with
            `_motionscape`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgFigure: the scape, with the track and the triangle in `.data`.
    """
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._utils import MgFigure

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_motionscape.png', target_name, overwrite)

    if track is not None:
        track = np.asarray(track, dtype=np.float64).ravel()
        if track_times is not None:
            times = np.asarray(track_times, dtype=np.float64).ravel()
            if len(times) != len(track):
                raise ValueError(
                    f"track_times has {len(times)} values and track has {len(track)}; "
                    "they must describe the same series")
        else:
            times = np.arange(len(track)) / float(self.fps)
        used = "supplied track"
    else:
        views = motion_vector_views(self.filename)
        track, times, used = views.magnitude, views.time, "motion vectors"

    scape = motion_scape(track, n_scales=n_scales)
    shown = scape / np.nanpercentile(scape, 99) if np.isfinite(scape).any() else scape
    shown = np.power(np.clip(shown, 0, 1), gamma)

    minutes = (times[-1] / 60) if len(times) else 1
    fig, ax = plt.subplots(figsize=(14, 7), dpi=120)
    ax.imshow(shown, cmap=colormap, aspect="auto", interpolation="nearest",
              extent=(0, minutes, 0, n_scales))
    ax.set_xlabel("minutes")
    ax.set_ylabel("window length: short at the base, whole recording at the apex")
    ax.set_yticks([])
    ax.set_title(f"{os.path.basename(self.filename)} --- motion scape ({used})")
    fig.tight_layout()
    fig.savefig(target_name)
    plt.close(fig)

    self.motionscape_figure = MgFigure(
        figure=None, figure_type="video.motionscape",
        data={"track": track, "time": times, "scape": scape, "source": used},
        layers=None, image=target_name)
    return self.motionscape_figure