Video

Bases: MgAudio

This is the class for working with video files in the Musical Gestures Toolbox. It inherites from the class MgAudio for working with audio files as well. There is a set of preprocessing tools you can use when you load a video, such as: - trimming: to extract a section of the video, - skipping: to shrink the video by skipping N frames after keeping one, - rotating: to rotate the video by N degrees, - applying brightness and contrast - cropping: to crop the video either automatically (by assessing the area of motion) or manually with a pop-up user interface, - converting to grayscale

These preprocesses will apply upon creating the MgVideo. Further processes are available as class methods.

Initializes Musical Gestures data structure from a video file, and applies preprocesses if desired.

Parameters:

Name	Type	Description	Default
filename	Union[str, List[str]]	Path to the video file. If input is a list, will merge all videos into one.	required
array	ndarray	Generates an MgVideo object from a video array. Defaults to None.	None
fps	float	The frequency at which consecutive images from the video array are captured or displayed. Defaults to None.	None
path	str	Path to save the output video file generated from a video array. Defaults to None.	None
filtertype	str	The filtertype parameter for the motion() method. Regular turns all values below threshold to 0. Binary turns all values below threshold to 0, above threshold to 1. Blob removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	The threshold parameter for the motion() method. Eliminates pixel values less than given threshold. A number in the range of 0 to 1. Defaults to 0.05.	0.05
starttime	int or float	Trims the video from this start time (s). Defaults to 0.	0
endtime	int or float	Trims the video until this end time (s). Defaults to 0 (which means the full length).	0
blur	str	The blur parameter for the motion() method. 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
skip	int	Time-shrinks the video by skipping (discarding) every n frames determined by skip. Defaults to 0.	0
frames	int	Specify a fixed target number of frames to extract from the video. Defaults to 0.	0
rotate	int	Rotates the video by a rotate degrees. Defaults to 0.	0
color	bool	If False, converts the video to grayscale and sets every method in grayscale mode. Defaults to True.	True
contrast	int	Applies +/- 100 contrast to video. Defaults to 0.	0
brightness	int	Applies +/- 100 brightness to video. Defaults to 0.	0
crop	str	If 'manual', opens a window displaying the first frame of the input video file, where the user can draw a rectangle to which cropping is applied. If 'auto' the cropping function attempts to determine the area of significant motion and applies the cropping to that area. Defaults to 'None'.	'None'
keep_all	bool	If True, preserves an output video file after each used preprocessing stage. Defaults to False.	False
returned_by_process	bool	This parameter is only for internal use, do not use it. Defaults to False.	False
sr	int	Sampling rate of the audio file. Defaults to 22050.	22050
n_fft	int	Length of the FFT window. Defaults to 2048.	2048
hop_length	int	Number of samples between successive frames. Defaults to 512.	512

Source code in musicalgestures/_video.py

def __init__(
    self,
    filename: Union[str, List[str]],
    array=None,
    fps: float | None = None,
    path: str | None = None,
    # Video parameters
    filtertype: str = "Regular",
    threshold: float = 0.05,
    starttime: float = 0,
    endtime: float = 0,
    blur: str = "None",
    skip: int = 0,
    frames: int = 0,
    rotate: float = 0,
    color: bool = True,
    contrast: float = 0,
    brightness: float = 0,
    crop: str = "None",
    keep_all: bool = False,
    returned_by_process: bool = False,
    # Audio parameters
    sr: int = 22050,
    n_fft: int = 2048,
    hop_length: int = 512,
):
    """
    Initializes Musical Gestures data structure from a video file, and applies preprocesses if desired.

    Args:
        filename (Union[str, List[str]]): Path to the video file. If input is a list, will merge all videos into one.
        array (np.ndarray, optional): Generates an MgVideo object from a video array. Defaults to None.
        fps (float, optional): The frequency at which consecutive images from the video array are captured or displayed. Defaults to None.
        path (str, optional): Path to save the output video file generated from a video array. Defaults to None.
        filtertype (str, optional): The `filtertype` parameter for the `motion()` method. `Regular` turns all values below `threshold` to 0. `Binary` turns all values below `threshold` to 0, above `threshold` to 1. `Blob` removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): The `threshold` parameter for the `motion()` method. Eliminates pixel values less than given threshold. A number in the range of 0 to 1. Defaults to 0.05.
        starttime (int or float, optional): Trims the video from this start time (s). Defaults to 0.
        endtime (int or float, optional): Trims the video until this end time (s). Defaults to 0 (which means the full length).
        blur (str, optional): The `blur` parameter for the `motion()` method. 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        skip (int, optional): Time-shrinks the video by skipping (discarding) every n frames determined by `skip`. Defaults to 0.
        frames (int, optional): Specify a fixed target number of frames to extract from the video. Defaults to 0.
        rotate (int, optional): Rotates the video by a `rotate` degrees. Defaults to 0.
        color (bool, optional): If False, converts the video to grayscale and sets every method in grayscale mode. Defaults to True.
        contrast (int, optional): Applies +/- 100 contrast to video. Defaults to 0.
        brightness (int, optional): Applies +/- 100 brightness to video. Defaults to 0.
        crop (str, optional): If 'manual', opens a window displaying the first frame of the input video file, where the user can draw a rectangle to which cropping is applied. If 'auto' the cropping function attempts to determine the area of significant motion and applies the cropping to that area. Defaults to 'None'.
        keep_all (bool, optional): If True, preserves an output video file after each used preprocessing stage. Defaults to False.
        returned_by_process (bool, optional): This parameter is only for internal use, do not use it. Defaults to False.

        sr (int, optional): Sampling rate of the audio file. Defaults to 22050.
        n_fft (int, optional): Length of the FFT window. Defaults to 2048.
        hop_length (int, optional): Number of samples between successive frames. Defaults to 512.
    """

    # if filename is a list, merge all videos into one
    if isinstance(filename, list):
        self.filename = merge_videos(filename)
    else:
        self.filename = filename

    self.array = array
    # The ARGUMENT is optional: it is the rate an array is encoded at, and a file
    # carries its own. The ATTRIBUTE is not, because `get_video()` below reads the
    # true rate from the file and overwrites this, so by the end of `__init__` it is
    # always a number. 0.0 stands for "not given yet" over the few lines before that
    # happens, and the from-array branch below tests it for truth rather than for
    # None, which rejects an explicit `fps=0` --- not a frame rate anything can use.
    self.fps: float = fps if fps is not None else 0.0
    self.path = path
    # Name of file without extension (only-filename)
    self.of = os.path.splitext(self.filename)[0]
    self.fex = os.path.splitext(self.filename)[1]
    # Video parameters
    self.color = color
    self.starttime = starttime
    self.endtime = endtime
    self.skip = skip
    self.frames = frames
    self.filtertype = filtertype
    self.threshold = threshold
    self.blur = blur
    self.contrast = contrast
    self.brightness = brightness
    self.crop = crop
    self.rotate = rotate
    self.keep_all = keep_all
    self.has_audio = None
    self.returned_by_process = returned_by_process
    # Audio parameters
    self.sr = sr
    self.n_fft = n_fft
    self.hop_length = hop_length

    # Check input and if FFmpeg is properly installed
    self.test_input()

    if self.array is not None and not self.fps:
        # Without a rate there is nothing to encode the array at, so the write below
        # never happens and the read after it fails with "no such file" --- blaming
        # the output for a missing argument. Both before and after this attribute
        # stopped being Optional, so say what is actually wrong.
        raise ValueError(
            "fps= is required when MgVideo is given an array: an array carries no frame "
            "rate of its own, so there is nothing to encode it at.")

    if self.array is not None and self.fps:
        self.from_numpy(self.array, self.fps)
    elif fps is not None:
        # `fps` is the rate an ARRAY is encoded at. For a file input `get_video()` below reads
        # the rate from the file and overwrites this unconditionally, so the argument does
        # nothing --- and code written on the belief that it had set the rate would be wrong
        # with nothing in any output to show it. Say so rather than let it pass in silence.
        warnings.warn(
            "fps= is ignored when MgVideo is given a file: the rate is read from the file "
            "itself, so this argument has no effect. Use resample(fps=...) to change a "
            "file's rate, which returns a new MgVideo that has re-read it.",
            UserWarning, stacklevel=2)

    self.get_video()
    self.flow = Flow(self, self.filename, self.color, self.has_audio)

filename instance-attribute

filename = merge_videos(filename)

array instance-attribute

array = array

fps instance-attribute

fps = fps if fps is not None else 0.0

path instance-attribute

path = path

of instance-attribute

of = os.path.splitext(self.filename)[0]

fex instance-attribute

fex = os.path.splitext(self.filename)[1]

color instance-attribute

color = color

starttime instance-attribute

starttime = starttime

endtime instance-attribute

endtime = endtime

skip instance-attribute

skip = skip

frames instance-attribute

frames = frames

filtertype instance-attribute

filtertype = filtertype

threshold instance-attribute

threshold = threshold

blur instance-attribute

blur = blur

contrast instance-attribute

contrast = contrast

brightness instance-attribute

brightness = brightness

crop instance-attribute

crop = crop

rotate instance-attribute

rotate = rotate

keep_all instance-attribute

keep_all = keep_all

has_audio instance-attribute

has_audio = None

returned_by_process instance-attribute

returned_by_process = returned_by_process

sr instance-attribute

sr = sr

n_fft instance-attribute

n_fft = n_fft

hop_length instance-attribute

hop_length = hop_length

flow instance-attribute

flow = Flow(self, self.filename, self.color, self.has_audio)

_avg_frame_cache instance-attribute

_avg_frame_cache

_pose_keypoints instance-attribute

_pose_keypoints

as_avi instance-attribute

as_avi

as_mp4 instance-attribute

as_mp4

average_pose instance-attribute

average_pose

blend_image instance-attribute

blend_image

blur_faces_video instance-attribute

blur_faces_video

motion_audio_coupling_figure instance-attribute

motion_audio_coupling_figure

dynamics_coupling_figure instance-attribute

dynamics_coupling_figure

eulerian_video instance-attribute

eulerian_video

flow_dense_video instance-attribute

flow_dense_video

flow_sparse_video instance-attribute

flow_sparse_video

frameaverage_cv2_image instance-attribute

frameaverage_cv2_image

frameaverage_image instance-attribute

frameaverage_image

heatmap_image instance-attribute

heatmap_image

history_video instance-attribute

history_video

mhi_image instance-attribute

mhi_image

motion_video instance-attribute

motion_video

motiondescriptors_figure instance-attribute

motiondescriptors_figure

actions instance-attribute

actions

postures instance-attribute

postures

motion_plot_image instance-attribute

motion_plot_image

motiongram_y_image instance-attribute

motiongram_y_image

motiongram_x_image instance-attribute

motiongram_x_image

motionvectors_video instance-attribute

motionvectors_video

motionvectorhistory_image instance-attribute

motionvectorhistory_image

motionvectorgrams_images instance-attribute

motionvectorgrams_images

motionvectorwaterfall_figure instance-attribute

motionvectorwaterfall_figure

motionvectoroverview_figure instance-attribute

motionvectoroverview_figure

motionscape_figure instance-attribute

motionscape_figure

posegram_figure instance-attribute

posegram_figure

painting_figure instance-attribute

painting_figure

colourgram_image instance-attribute

colourgram_image

eyetracking_figure instance-attribute

eyetracking_figure

gazegram_x_image instance-attribute

gazegram_x_image

gazegram_y_image instance-attribute

gazegram_y_image

posegram_spatial_figure instance-attribute

posegram_spatial_figure

posegrams_images instance-attribute

posegrams_images

movement_beat_statistics_figure instance-attribute

movement_beat_statistics_figure

phase_synchrony_figure instance-attribute

phase_synchrony_figure

pose_average_image instance-attribute

pose_average_image

pose_centered_figure instance-attribute

pose_centered_figure

pose_distance_figure instance-attribute

pose_distance_figure

pose_segments_figure instance-attribute

pose_segments_figure

pose_trajectories_image instance-attribute

pose_trajectories_image

pose_video instance-attribute

pose_video

pose_waterfall_figure instance-attribute

pose_waterfall_figure

silhouette_waterfall_figure instance-attribute

silhouette_waterfall_figure

sonomotiongram_audio instance-attribute

sonomotiongram_audio

spacetime_volume_figure instance-attribute

spacetime_volume_figure

ssm_combined_image instance-attribute

ssm_combined_image

stroboscope_image instance-attribute

stroboscope_image

multishot_image instance-attribute

multishot_image

pose_timeline_figure instance-attribute

pose_timeline_figure

plate_image instance-attribute

plate_image

structure_comparison_figure instance-attribute

structure_comparison_figure

subtract_video instance-attribute

subtract_video

tempo_similarity_figure instance-attribute

tempo_similarity_figure

trajectories instance-attribute

trajectories

videogram_y_image instance-attribute

videogram_y_image

videogram_x_image instance-attribute

videogram_x_image

warp_video instance-attribute

warp_video

motion_plot class-attribute instance-attribute

motion_plot = deprecated_alias('motion_plot', 'motion_plot_image')

motiongram_x class-attribute instance-attribute

motiongram_x = deprecated_alias('motiongram_x', 'motiongram_x_image')

motiongram_y class-attribute instance-attribute

motiongram_y = deprecated_alias('motiongram_y', 'motiongram_y_image')

motiongram_vertical_image class-attribute instance-attribute

motiongram_vertical_image = deprecated_alias('motiongram_vertical_image', 'motiongram_x_image')

motiongram_horizontal_image class-attribute instance-attribute

motiongram_horizontal_image = deprecated_alias('motiongram_horizontal_image', 'motiongram_y_image')

movement_beat_statistics class-attribute instance-attribute

movement_beat_statistics = deprecated_alias('movement_beat_statistics', 'movement_beat_statistics_figure')

body_audio_coupling_figure class-attribute instance-attribute

body_audio_coupling_figure = deprecated_alias('body_audio_coupling_figure', 'motion_audio_coupling_figure')

pose_average class-attribute instance-attribute

pose_average = deprecated_alias('pose_average', 'pose_average_image')

pose_trajectories class-attribute instance-attribute

pose_trajectories = deprecated_alias('pose_trajectories', 'pose_trajectories_image')

ssm_combined class-attribute instance-attribute

ssm_combined = deprecated_alias('ssm_combined', 'ssm_combined_image')

videogram_x class-attribute instance-attribute

videogram_x = deprecated_alias('videogram_x', 'videogram_x_image')

videogram_y class-attribute instance-attribute

videogram_y = deprecated_alias('videogram_y', 'videogram_y_image')

videogram_vertical_image class-attribute instance-attribute

videogram_vertical_image = deprecated_alias('videogram_vertical_image', 'videogram_x_image')

videogram_horizontal_image class-attribute instance-attribute

videogram_horizontal_image = deprecated_alias('videogram_horizontal_image', 'videogram_y_image')

n_frames property

n_frames

Number of frames in the video (an alias for the frame-count length).

duration property

duration

Video duration in seconds (length / fps).

Note self.length is the frame count for an MgVideo (it is the duration in seconds for an MgAudio); use this property when you want seconds.

motion_mp

motion_mp(*args, **kwargs)

Retired. Use :meth:motion instead.

This rendered a motion video across several processes, coordinating them with a socket server and an argparse child process. It raised on its first call --- two fields were read as one between the parent and the workers, and behind that failure it called save_txt and save_analysis with arguments in the wrong positions --- so it produced nothing for anyone who tried it. See issue #370.

motion() does the same work in one process and is tested.

Raises:

Type	Description
NotImplementedError	Always.

Source code in musicalgestures/_video.py

def motion_mp(self, *args, **kwargs):
    """Retired. Use :meth:`motion` instead.

    This rendered a motion video across several processes, coordinating them
    with a socket server and an argparse child process. It raised on its first
    call --- two fields were read as one between the parent and the workers,
    and behind that failure it called `save_txt` and `save_analysis` with
    arguments in the wrong positions --- so it produced nothing for anyone who
    tried it. See issue #370.

    `motion()` does the same work in one process and is tested.

    Raises:
        NotImplementedError: Always.
    """
    raise NotImplementedError(
        "motion_mp() has been retired: it raised on its first call and produced "
        "nothing (see issue #370). Use motion(), which does the same work in one "
        "process.")

warp_audiovisual_beats

warp_audiovisual_beats(*args, **kwargs)

Retired name: use :meth:warp_audiomotion_beats.

Source code in musicalgestures/_video.py

def warp_audiovisual_beats(self, *args, **kwargs):
    """Retired name: use :meth:`warp_audiomotion_beats`."""
    warnings.warn("warp_audiovisual_beats() is deprecated and will be removed in 2.0; "
                  "use warp_audiomotion_beats().", DeprecationWarning, stacklevel=2)
    return self.warp_audiomotion_beats(*args, **kwargs)

body_audio_coupling

body_audio_coupling(*args, **kwargs)

Retired name: use :meth:motion_audio_coupling.

Source code in musicalgestures/_video.py

def body_audio_coupling(self, *args, **kwargs):
    """Retired name: use :meth:`motion_audio_coupling`."""
    warnings.warn("body_audio_coupling() is deprecated and will be removed in 2.0; "
                  "use motion_audio_coupling().", DeprecationWarning, stacklevel=2)
    return self.motion_audio_coupling(*args, **kwargs)

motion

motion(filtertype='Regular', threshold=0.05, blur='None', kernel_size=5, use_median=False, unit='seconds', atadenoise=False, motion_analysis='all', inverted_motionvideo=False, inverted_motiongram=False, equalize_motiongram=False, audio_descriptors=False, save_plot=True, title=None, save_data=True, data_format='csv', save_motiongrams=True, save_video=True, target_name_video=None, target_name_plot=None, target_name_data=None, target_name_mgx=None, target_name_mgy=None, overwrite=True, normalize=None)

Finds the difference in pixel value from one frame to the next in an input video, and saves the frames into a new video. Describes the motion in the recording. Outputs: a motion video, a plot describing the centroid of motion and the quantity of motion, the y- and x-motiongrams, and a text file containing the quantity of motion and the centroid of motion for each frame with timecodes in milliseconds.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
kernel_size	int	Size of structuring element. Defaults to 5.	5
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
unit	str	Unit in QoM plot. Accepted values are 'seconds' or 'samples'. Defaults to 'seconds'.	'seconds'
atadenoise	bool	If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.	False
normalize	bool | None	Chooses between data for looking at and data for measuring with. True (and None, the long-standing default) writes a Qom column divided by that clip's own maximum, so every clip peaks at 1.0 -- what a plot on a 0-1 axis wants, and the reason the value exists. It also makes quantity of motion incomparable between clips and lets one bright frame set the scale. False writes QomRaw, the untouched sum of pixel values in the thresholded difference frame: comparable across clips, on the scale the pixels had. The column is renamed rather than rescaled so a file cannot be misread later.	None
motion_analysis	str	Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.	'all'
inverted_motionvideo	bool	If True, inverts colors of the motion video. Defaults to False.	False
inverted_motiongram	bool	If True, inverts colors of the motiongrams. Defaults to False.	False
equalize_motiongram	bool	If True, converts the motiongrams to hsv-color space and flattens the value channel (v). Defaults to True.	False
save_plot	bool	If True, outputs motion-plot. Defaults to True.	True
title	str	Optionally add title to the plot. Defaults to None, which uses the file name as a title.	None
save_data	bool	If True, outputs motion-data. Defaults to True.	True
data_format	str / list	Specifies format of motion-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.	'csv'
save_motiongrams	bool	If True, outputs motiongrams. Defaults to True.	True
save_video	bool	If True, outputs the motion video. Defaults to True.	True
target_name_video	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).	None
target_name_plot	str	Target output name for the plot. Defaults to None (which assumes that the input filename with the suffix "_motion_com_aom_qom" should be used).	None
target_name_data	str	Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).	None
target_name_mgx	str	Target output name for the x-motiongram (the tall picture). Defaults to None (which assumes that the input filename with the suffix "_mgv" should be used).	None
target_name_mgy	str	Target output name for the y-motiongram (the wide picture). Defaults to None (which assumes that the input filename with the suffix "_mgh" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	'musicalgestures.MgVideo'	A new MgVideo pointing to the output video file. If save_video=False, it returns an MgVideo pointing to the input video file.

Source code in musicalgestures/_motionvideo.py

def mg_motion(
        self: "musicalgestures.MgVideo",
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        kernel_size: int = 5,
        use_median: bool = False,
        unit: str = 'seconds',
        atadenoise: bool = False,
        motion_analysis: str = 'all',
        inverted_motionvideo: bool = False,
        inverted_motiongram: bool = False,
        equalize_motiongram: bool = False,
        audio_descriptors: bool = False,
        save_plot: bool = True,
        title: str | None = None,
        save_data: bool = True,
        data_format: str | list = "csv",
        save_motiongrams: bool = True,
        save_video: bool = True,
        target_name_video: str | None = None,
        target_name_plot: str | None = None,
        target_name_data: str | None = None,
        target_name_mgx: str | None = None,
        target_name_mgy: str | None = None,
        overwrite: bool = True,
        normalize: bool | None = None) -> "musicalgestures.MgVideo":
    """
    Finds the difference in pixel value from one frame to the next in an input video, and saves the frames into a new video. 
    Describes the motion in the recording. Outputs: a motion video, a plot describing the centroid of motion and the 
    quantity of motion, the y- and x-motiongrams, and a text file containing the quantity of motion and the 
    centroid of motion for each frame with timecodes in milliseconds.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        unit (str, optional): Unit in QoM plot. Accepted values are 'seconds' or 'samples'. Defaults to 'seconds'.
        atadenoise (bool, optional): If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.
        normalize (bool | None, optional): Chooses between data for looking at and data for
            measuring with. True (and None, the long-standing default) writes a `Qom` column
            divided by that clip's own maximum, so every clip peaks at 1.0 -- what a plot on a
            0-1 axis wants, and the reason the value exists. It also makes quantity of motion
            **incomparable between clips** and lets one bright frame set the scale. False writes
            `QomRaw`, the untouched sum of pixel values in the thresholded difference frame:
            comparable across clips, on the scale the pixels had. The column is renamed rather
            than rescaled so a file cannot be misread later.
        motion_analysis (str, optional): Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.
        inverted_motionvideo (bool, optional): If True, inverts colors of the motion video. Defaults to False.
        inverted_motiongram (bool, optional): If True, inverts colors of the motiongrams. Defaults to False.
        equalize_motiongram (bool, optional): If True, converts the motiongrams to hsv-color space and flattens the value channel (v). Defaults to True.
        save_plot (bool, optional): If True, outputs motion-plot. Defaults to True.
        title (str, optional): Optionally add title to the plot. Defaults to None, which uses the file name as a title.
        save_data (bool, optional): If True, outputs motion-data. Defaults to True.
        data_format (str/list, optional): Specifies format of motion-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.
        save_motiongrams (bool, optional): If True, outputs motiongrams. Defaults to True.
        save_video (bool, optional): If True, outputs the motion video. Defaults to True.
        target_name_video (str, optional): Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).
        target_name_plot (str, optional): Target output name for the plot. Defaults to None (which assumes that the input filename with the suffix "_motion_com_aom_qom" should be used).
        target_name_data (str, optional): Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).
        target_name_mgx (str, optional): Target output name for the x-motiongram (the tall picture). Defaults to None (which assumes that the input filename with the suffix "_mgv" should be used).
        target_name_mgy (str, optional): Target output name for the y-motiongram (the wide picture). Defaults to None (which assumes that the input filename with the suffix "_mgh" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A new MgVideo pointing to the output video file. If `save_video=False`, it returns an MgVideo pointing to the input video file.
    """

    if save_plot | save_data | save_motiongrams | save_video:
        # ignore runtime warnings when dividing by 0
        np.seterr(divide='ignore', invalid='ignore')

        of, fex = self.of, self.fex

        # Define ffmpeg command start and end
        cmd = ['ffmpeg', '-y', '-i', self.filename]
        # Filter video frames using ffmpeg
        cmd, cmd_filter = filter_frame_ffmpeg(self.filename, cmd, self.color, blur, filtertype, threshold, kernel_size, use_median)

        if atadenoise:
            # Apply an adaptive temporal averaging denoiser every 129 frames
            cmd_filter += 'atadenoise=s=129'
        else:
            # Remove last comma after previous filter
            cmd_filter = cmd_filter[: -1]
        cmd += ['-filter_complex', cmd_filter] 

        if save_motiongrams:
            #: Columns are collected in a list and concatenated ONCE after the loop.
            #: `np.append` in a per-frame loop reallocates the whole gram every frame,
            #: which is O(n^2) in frames and is what made a long recording impossible:
            #: measured 69 s / 148 s / 366 s for 30 / 60 / 120 s of 1080p video, an
            #: exponent rising from 1.10 to 1.31, extrapolating to about 215 hours for
            #: a 2 h 38 min file. The values are unchanged; only the allocation is.
            gramx_parts = [np.zeros([1, self.width, 3]).astype(np.uint8)]
            gramy_parts = [np.zeros([self.height, 1, 3]).astype(np.uint8)] 

        if save_data | save_plot:
            #: Lists, converted to arrays once after the loop, for the reason above.
            time_l: list = []   # time in ms
            aom_l: list = []    # area of motion
            qom_l: list = []    # quantity of motion
            com_l: list = []    # centroid of motion

        if save_video:
            if target_name_video is None:
                target_name_video = of + '_motion' + fex
            # enforce avi
            else:
                target_name_video = os.path.splitext(target_name_video)[0] + fex
            if not overwrite:
                target_name_video = generate_outfilename(target_name_video)

        pgbar_text = 'Rendering motion' + ", ".join(np.array(["-video", "-grams", "-plots", "-data"])[
            np.array([save_video, save_motiongrams, save_plot, save_data])]) + ":" 
        pb = MgProgressbar(total=self.length, prefix=pgbar_text)  

        # Pipe video with FFmpeg for reading frame by frame        
        process = ffmpeg_cmd(cmd, total_time=self.length, pipe='read')
        video_out = None

        i = 0
        while True:
            # Read frame-by-frame
            out = process.stdout.read(self.width*self.height*3)

            if out == b'':
                pb.progress(self.length)
                break

            # Transform the bytes read into a numpy array
            motion_frame = np.frombuffer(out, dtype=np.uint8).reshape([self.height, self.width, 3]) # height, width, channels

            if save_data | save_plot:
                if motion_analysis.lower() == 'aom':
                    # Area of Motion (AoM)
                    aombite = area(motion_frame, self.height, self.width)
                    time_l.append(frame2ms(i, self.fps))
                    aom_l.append(np.asarray(aombite).reshape(4))

                if motion_analysis.lower() == 'com' or motion_analysis.lower() == 'qom':
                    # Centroid of Motion (CoM) and Quantity of Motion (QoM)
                    combite, qombite = centroid(motion_frame, self.width, self.height)
                    time_l.append(frame2ms(i, self.fps))
                    com_l.append(np.asarray(combite).reshape(2))
                    qom_l.append(qombite)

                if motion_analysis.lower() == 'all':
                    # Area of Motion (AoM)
                    aombite = area(motion_frame, self.height, self.width)
                    # Centroid of Motion (CoM) and Quantity of Motion (QoM)
                    combite, qombite = centroid(motion_frame, self.width, self.height)
                    time_l.append(frame2ms(i, self.fps))
                    com_l.append(np.asarray(combite).reshape(2))
                    qom_l.append(qombite)
                    aom_l.append(np.asarray(aombite).reshape(4))

            if save_motiongrams:
                movement_y = np.rint(np.mean(motion_frame, axis=1)).reshape(self.height, 1, 3).astype(np.uint8)
                movement_x = np.rint(np.mean(motion_frame, axis=0)).reshape(1, self.width, 3).astype(np.uint8)

                gramy_parts.append(movement_y)
                gramx_parts.append(movement_x)

            if save_video:
                if video_out is None:
                    assert target_name_video is not None  # resolved under `if save_video` above
                    cmd =['ffmpeg', '-y', '-s', '{}x{}'.format(motion_frame.shape[1], motion_frame.shape[0]), 
                        '-r', str(self.fps), '-f', 'rawvideo', '-pix_fmt', 'bgr24', '-vcodec', 'rawvideo', 
                        '-i', '-', '-vcodec', 'libx264', '-pix_fmt', 'yuv420p', target_name_video]
                    video_out = ffmpeg_cmd(cmd, total_time=self.length, pipe='write')

                if inverted_motionvideo:
                    video_out.stdin.write(cv2.bitwise_not(motion_frame.astype(np.uint8)))
                else:
                    video_out.stdin.write(motion_frame.astype(np.uint8))

            # Flush the buffer
            process.stdout.flush()
            pb.progress(i)
            i += 1

        # Terminate the processes
        if save_video:
            # the writer opens on the first frame, so this only holds for a video that had frames
            assert video_out is not None, "no frames were written: the video decoded to nothing"
            video_out.stdin.close()
            video_out.wait()
        process.terminate()

        #: One allocation each, now that every frame has been seen. Empty inputs keep
        #: the shapes the rest of this function has always expected.
        if save_data | save_plot:
            time = np.array(time_l)
            qom = np.array(qom_l)
            com = np.array(com_l) if com_l else np.array([])
            aom = np.array(aom_l) if aom_l else np.array([])
        if save_motiongrams:
            gramx = np.concatenate(gramx_parts, axis=0).astype(np.uint8)
            gramy = np.concatenate(gramy_parts, axis=1).astype(np.uint8)

        if save_motiongrams:
            gramx = (gramx-gramx.min())/(gramx.max()-gramx.min())*255.0
            gramy = (gramy-gramy.min())/(gramy.max()-gramy.min())*255.0

            if equalize_motiongram:
                gramx = gramx.astype(np.uint8)
                gramx_hsv = cv2.cvtColor(gramx, cv2.COLOR_RGB2HSV).astype(np.uint8)
                gramx_hsv[:, :, 2] = cv2.equalizeHist(gramx_hsv[:, :, 2]).astype(np.uint8)
                gramx = cv2.cvtColor(gramx_hsv, cv2.COLOR_HSV2RGB).astype(np.uint8)

                gramy = gramy.astype(np.uint8)
                gramy_hsv = cv2.cvtColor(gramy, cv2.COLOR_RGB2HSV).astype(np.uint8)
                gramy_hsv[:, :, 2] = cv2.equalizeHist(gramy_hsv[:, :, 2]).astype(np.uint8)
                gramy = cv2.cvtColor(gramy_hsv, cv2.COLOR_HSV2RGB).astype(np.uint8)

            if target_name_mgx is None:
                target_name_mgx = of + '_mgv.png'
            if target_name_mgy is None:
                target_name_mgy = of + '_mgh.png'
            if not overwrite:
                target_name_mgx = generate_outfilename(target_name_mgx)
                target_name_mgy = generate_outfilename(target_name_mgy)

            if inverted_motiongram:
                cv2.imwrite(target_name_mgx, cv2.bitwise_not(gramx.astype(np.uint8)))
                cv2.imwrite(target_name_mgy, cv2.bitwise_not(gramy.astype(np.uint8)))
            else:
                cv2.imwrite(target_name_mgx, gramx.astype(np.uint8))
                cv2.imwrite(target_name_mgy, gramy.astype(np.uint8))

            # save motiongrams data and convert to grayscale for processing motiongrams Self-Similarity Matrices (SSMs)
            data = (cv2.cvtColor(gramx.astype(np.uint8), cv2.COLOR_RGB2GRAY), cv2.cvtColor(gramy.astype(np.uint8), cv2.COLOR_RGB2GRAY))
            self.ssm_figure = MgFigure(figure=None, figure_type='video.ssm', data=data, layers=None, image=(target_name_mgx, target_name_mgy))

            # save rendered motiongrams as MgImages into parent MgVideo
            self.motiongram_x_image = MgImage(target_name_mgx)
            self.motiongram_y_image = MgImage(target_name_mgy)

        # `audio_descriptors` arrives as a flag and travels on as the video to read
        # audio from, which save_analysis expects. Two meanings, two names.
        audio_descriptors_source: "bool | musicalgestures.MgVideo" = audio_descriptors
        if audio_descriptors:
            audio_descriptors_source = self

        if save_data:
            # `normalize` was declared on mg_motion and never used. It now
            # chooses between the per-clip normalised `Qom` column, which is
            # what a plot wants, and the raw `QomRaw` sum of pixel values,
            # which is what a cross-clip measurement needs. None keeps the
            # long-standing default so no existing call changes.
            save_txt(of, time, aom, com, qom, motion_analysis, self.width, self.height, 
            data_format=data_format, target_name_data=target_name_data, overwrite=overwrite,
            normalize=True if normalize is None else bool(normalize))

        if save_plot:
            if title is None:
                title = os.path.basename(of + fex)
            # save plot as an MgImage at motion_plot_image for parent MgVideo
            self.motion_plot_image = MgImage(save_analysis(of, self.fps, aom, com, qom, motion_analysis, audio_descriptors_source, self.width,
                                        self.height, unit, title, target_name_plot=target_name_plot, overwrite=overwrite))

        # Resetting numpy warnings for dividing by 0
        np.seterr(divide='warn', invalid='warn')

        if save_video:
            # Check if the original video fil has audio
            if self.has_audio:
                source_audio = extract_wav(of + fex)
                embed_audio_in_video(source_audio, target_name_video)
                os.remove(source_audio)

            # Save generated musicalgestures video as the video of the parent MgVideo
            assert target_name_video is not None  # resolved under `if save_video` above
            self.motion_video = musicalgestures.MgVideo(filename=target_name_video, returned_by_process=True)
            return self.motion_video

        else:
            return self

    else:
        # Return the MgVideo the motion() was called upon
        print("Nothing to render. Exiting...")
        return self

motiongrams

motiongrams(filtertype='Regular', threshold=0.05, blur='None', use_median=False, atadenoise=False, kernel_size=5, inverted_motiongram=False, equalize_motiongram=True, target_name_mgx=None, target_name_mgy=None, overwrite=True, normalize=None)

Shortcut for mg_motion to only render motiongrams.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
atadenoise	bool	If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.	False
kernel_size	int	Size of the median filter (if use_median=True) or the erosion filter (if filtertype='blob'). Defaults to 5.	5
inverted_motiongram	bool	If True, inverts colors of the motiongrams. Defaults to False.	False
equalize_motiongram	bool	If True, converts the motiongrams to hsv-color space and flattens the value channel (v). Defaults to True.	True
target_name_mgx	str	Target output name for the x-motiongram (the tall picture). Defaults to None (which assumes that the input filename with the suffix "_mgv" should be used).	None
target_name_mgy	str	Target output name for the y-motiongram (the wide picture). Defaults to None (which assumes that the input filename with the suffix "_mgh" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgList	'MgList'	An MgList pointing to the output motiongram images (as MgImages).

Source code in musicalgestures/_motionvideo.py

def mg_motiongrams(
        self: "musicalgestures.MgVideo",
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        use_median: bool = False,
        atadenoise: bool = False,
        kernel_size: int = 5,
        inverted_motiongram: bool = False,
        equalize_motiongram: bool = True,
        target_name_mgx: str | None = None,
        target_name_mgy: str | None = None,
        overwrite: bool = True,
        normalize: bool | None = None) -> "MgList":
    """
    Shortcut for `mg_motion` to only render motiongrams.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        atadenoise (bool, optional): If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.
        kernel_size (int, optional): Size of the median filter (if `use_median=True`) or the erosion filter (if `filtertype='blob'`). Defaults to 5.
        inverted_motiongram (bool, optional): If True, inverts colors of the motiongrams. Defaults to False.
        equalize_motiongram (bool, optional): If True, converts the motiongrams to hsv-color space and flattens the value channel (v). Defaults to True.
        target_name_mgx (str, optional): Target output name for the x-motiongram (the tall picture). Defaults to None (which assumes that the input filename with the suffix "_mgv" should be used).
        target_name_mgy (str, optional): Target output name for the y-motiongram (the wide picture). Defaults to None (which assumes that the input filename with the suffix "_mgh" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgList: An MgList pointing to the output motiongram images (as MgImages).
    """

    if target_name_mgx is None:
        target_name_mgx = self.of + '_mgv.png'
    if target_name_mgy is None:
        target_name_mgy = self.of + '_mgh.png'
    if not overwrite:
        out_x = generate_outfilename(target_name_mgx)
        out_y = generate_outfilename(target_name_mgy)
    else:
        out_x, out_y = target_name_mgx, target_name_mgy

    mg_motion(
        self,
        filtertype=filtertype,
        threshold=threshold,
        blur=blur,
        kernel_size=kernel_size,
        use_median=use_median,
        atadenoise=atadenoise,
        inverted_motiongram=inverted_motiongram,
        equalize_motiongram=equalize_motiongram,
        save_data=False,
        save_motiongrams=True,
        save_plot=False,
        save_video=False,
        target_name_mgx=out_x,
        target_name_mgy=out_y,
        overwrite=True)

    # mg_motion also saves the motiongrams as MgImages to
    # self.motiongram_vertical_image and self.motiongram_horizontal_image of the parent MgVideo
    return MgList(MgImage(out_x), MgImage(out_y))

motiondata

motiondata(filtertype='Regular', threshold=0.05, blur='None', kernel_size=5, atadenoise=False, use_median=False, motion_analysis='all', data_format='csv', target_name=None, overwrite=True)

Shortcut for mg_motion to only render motion data.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
kernel_size	int	Size of structuring element. Defaults to 5.	5
atadenoise	bool	If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.	False
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
motion_analysis	str	Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.	'all'
data_format	str / list	Specifies format of motion-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.	'csv'
target_name	str	Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Type	Description
'str | list'	str/list: The path(s) to the rendered data file(s).

Source code in musicalgestures/_motionvideo.py

def mg_motiondata(
        self: "musicalgestures.MgVideo",
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        kernel_size: int = 5,
        atadenoise: bool = False,
        use_median: bool = False,
        motion_analysis: str = 'all',
        data_format: str | list = "csv",
        target_name: str | None = None,
        overwrite: bool = True) -> "str | list":
    """
    Shortcut for `mg_motion` to only render motion data.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        atadenoise (bool, optional): If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        motion_analysis (str, optional): Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.
        data_format (str/list, optional): Specifies format of motion-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.
        target_name (str, optional): Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        str/list: The path(s) to the rendered data file(s).
    """

    out: "str | list | None" = None

    if type(data_format) == str:
        if target_name is None:
            target_name = self.of + '_motion.' + data_format
        if not overwrite:
            target_name = generate_outfilename(target_name)
        out = target_name

    if type(data_format) == list:
        out = []
        paths: list = out
        if target_name is None:
            # this csv is just a temporary placeholder, the correct extension is always enforced based on the data_format(s)
            target_name = self.of + '_motion.csv'
        target_name_of = os.path.splitext(target_name)[0]
        for item in data_format:
            if not overwrite:
                tmp_name = generate_outfilename(target_name_of + '.' + item)
            else:
                tmp_name = target_name_of + '.' + item
            paths.append(tmp_name)

    mg_motion(
        self,
        filtertype=filtertype,
        threshold=threshold,
        blur=blur,
        kernel_size=kernel_size,
        use_median=use_median,
        atadenoise=atadenoise,
        motion_analysis=motion_analysis,
        data_format=data_format,
        save_data=True,
        save_motiongrams=False,
        save_plot=False,
        save_video=False,
        target_name_data=target_name,
        overwrite=overwrite)

    # if type(data_format) == list:
    #     outlist = [self.of + '_motion.' + elem for elem in data_format]
    #     return outlist
    # else:
    #     return self.of + '_motion.' + data_format
    assert out is not None, f"unsupported data_format: {data_format!r}"
    return out

motionplots

motionplots(filtertype='Regular', threshold=0.05, blur='None', kernel_size=5, use_median=False, atadenoise=False, motion_analysis='all', audio_descriptors=False, unit='seconds', title=None, target_name=None, overwrite=True)

Shortcut for mg_motion to only render motion plots.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
kernel_size	int	Size of structuring element. Defaults to 5.	5
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
atadenoise	bool	If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.	False
motion_analysis	str	Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.	'all'
audio_descriptors	bool	Whether to plot motion plots together with audio descriptors in order to see possible correlations in the data. Defaults to False.	False
unit	str	Unit in QoM plot. Accepted values are 'seconds' or 'samples'. Defaults to 'seconds'.	'seconds'
title	str	Optionally add title to the plot. Defaults to None, which uses the file name as a title.	None
target_name	str	Target output name for the plot. Defaults to None (which assumes that the input filename with the suffix "_motion_com_aom_qom" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	An MgImage pointing to the exported image (png) of the motion plots.

Source code in musicalgestures/_motionvideo.py

def mg_motionplots(
        self: "musicalgestures.MgVideo",
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        kernel_size: int = 5,
        use_median: bool = False,
        atadenoise: bool = False,
        motion_analysis: str = 'all',
        audio_descriptors: bool = False,
        unit: str = 'seconds',
        title: str | None = None,
        target_name: str | None = None,
        overwrite: bool = True) -> "MgImage":
    """
    Shortcut for `mg_motion` to only render motion plots.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        atadenoise (bool, optional): If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.
        motion_analysis (str, optional): Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.
        audio_descriptors (bool, optional): Whether to plot motion plots together with audio descriptors in order to see possible correlations in the data. Defaults to False.
        unit (str, optional): Unit in QoM plot. Accepted values are 'seconds' or 'samples'. Defaults to 'seconds'.
        title (str, optional): Optionally add title to the plot. Defaults to None, which uses the file name as a title.
        target_name (str, optional): Target output name for the plot. Defaults to None (which assumes that the input filename with the suffix "_motion_com_aom_qom" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgImage: An MgImage pointing to the exported image (png) of the motion plots.
    """

    if target_name is None:
        target_name = self.of + '_motionplots.png'
    if not overwrite:
        target_name = generate_outfilename(target_name)

    mg_motion(
        self,
        filtertype=filtertype,
        threshold=threshold,
        blur=blur,
        kernel_size=kernel_size,
        use_median=use_median,
        unit=unit,
        atadenoise=atadenoise,
        motion_analysis=motion_analysis,
        audio_descriptors=audio_descriptors,
        save_data=False,
        save_motiongrams=False,
        save_plot=True,
        title=title,
        save_video=False,
        target_name_plot=target_name,
        overwrite=overwrite)

    # mg_motion also saves the plot as an MgImage to self.motion_plot_image of the parent MgVideo
    return MgImage(target_name)

motionvideo

motionvideo(filtertype='Regular', threshold=0.05, blur='None', use_median=False, kernel_size=5, inverted_motionvideo=False, target_name=None, overwrite=True)

Shortcut to only render the motion video. Uses musicalgestures._utils.motionvideo_ffmpeg. Note that this does not apply median filter by default. If you need it use use_median=True.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
kernel_size	int	Size of the median filter (if use_median=True) or the erosion filter (if filtertype='blob'). Defaults to 5.	5
inverted_motionvideo	bool	If True, inverts colors of the motion video. Defaults to False.	False
target_name	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	'musicalgestures.MgVideo'	A new MgVideo pointing to the output '_motion' video file.

Source code in musicalgestures/_motionvideo.py

def mg_motionvideo(
        self: "musicalgestures.MgVideo",
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        use_median: bool = False,
        kernel_size: int = 5,
        inverted_motionvideo: bool = False,
        target_name: str | None = None,
        overwrite: bool = True) -> "musicalgestures.MgVideo":
    """
    Shortcut to only render the motion video. Uses musicalgestures._utils.motionvideo_ffmpeg. Note that this does not apply median filter by default. If you need it use `use_median=True`.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        kernel_size (int, optional): Size of the median filter (if `use_median=True`) or the erosion filter (if `filtertype='blob'`). Defaults to 5.
        inverted_motionvideo (bool, optional): If True, inverts colors of the motion video. Defaults to False.
        target_name (str, optional): Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A new MgVideo pointing to the output '_motion' video file.
    """

    motionvideo = motionvideo_ffmpeg(
        filename=self.filename,
        color=self.color,
        filtertype=filtertype,
        threshold=threshold,
        blur=blur,
        use_median=use_median,
        kernel_size=kernel_size,
        invert=inverted_motionvideo,
        target_name=target_name,
        overwrite=overwrite)

    # save motion video as motion_video for parent MgVideo
    # we have to do this here since we are not using mg_motion (that would normally save the result itself)
    self.motion_video = musicalgestures.MgVideo(motionvideo, color=self.color, returned_by_process=True)

    return self.motion_video

motionscore

motionscore()

Computes the average VMAF motion score of the video using FFmpeg.

Returns:

Name	Type	Description
float	'float | None'	The average VMAF motion score, or None if unavailable.

Source code in musicalgestures/_motionvideo.py

def mg_motionscore(self: "musicalgestures.MgVideo") -> "float | None":
    """
    Computes the average VMAF motion score of the video using FFmpeg.

    Returns:
        float: The average VMAF motion score, or None if unavailable.
    """
    cmd = ['ffmpeg', '-i', self.filename, '-vf', 'vmafmotion', '-f', 'null', '-']
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
    out, _ = process.communicate()
    splitted = out.split('\n')

    for index, item in enumerate(splitted):
        if isinstance(item, str) and re.search('VMAF', item):
            vmafmotion = float(re.findall(r'\d+\.\d+', splitted[index].split("] ")[1])[0])
            return vmafmotion
    print('VMAF motion score is not available.')
    return None

subtract

subtract(color=True, filtertype=None, threshold=0.05, blur=False, curves=0.15, use_median=False, kernel_size=5, bg_img=None, bg_color='#000000', target_name=None, overwrite=True)

Renders background subtraction using ffmpeg.

Parameters:

Name	Type	Description	Default
color	bool	If False the input is converted to grayscale at the start of the process. This can significantly reduce render time. Defaults to True.	True
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	None
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	bool	Whether to apply a smartblur ffmpeg filter or not. Defaults to False.	False
curves	int	Apply curves and equalisation threshold filter to subtract the background. Ranges from 0 to 1. Defaults to 0.15.	0.15
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
kernel_size	int	Size of the median filter (if use_median=True) or the erosion filter (if filtertype='blob'). Defaults to 5.	5
bg_img	str	Path to a background image (.png) that needs to be subtracted from the video. If set to None, it uses an average image of all frames in the video. Defaults to None.	None
bg_color	str	Set the background color in the video file in hex value. Defaults to '#000000' (black).	'#000000'
target_name	str	Target output name for the subtracted video. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	MgVideo	A MgVideo pointing to the subtracted video, also stored as subtract_video on the parent MgVideo

Source code in musicalgestures/_subtract.py

def mg_subtract(
        self: "musicalgestures.MgVideo",
        color=True,
        filtertype=None,
        threshold=0.05,
        blur=False,
        curves=0.15,
        use_median=False,
        kernel_size=5,
        bg_img=None,
        bg_color='#000000',
        target_name=None,
        overwrite=True) -> "musicalgestures.MgVideo":
    """
    Renders background subtraction using ffmpeg. 

    Args:
        color (bool, optional): If False the input is converted to grayscale at the start of the process. This can significantly reduce render time. Defaults to True.
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (bool, optional): Whether to apply a smartblur ffmpeg filter or not. Defaults to False.
        curves (int, optional): Apply curves and equalisation threshold filter to subtract the background. Ranges from 0 to 1. Defaults to 0.15.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        kernel_size (int, optional): Size of the median filter (if `use_median=True`) or the erosion filter (if `filtertype='blob'`). Defaults to 5.
        bg_img (str, optional): Path to a background image (.png) that needs to be subtracted from the video. If set to None, it uses an average image of all frames in the video. Defaults to None.
        bg_color (str, optional): Set the background color in the video file in hex value. Defaults to '#000000' (black). 
        target_name (str, optional): Target output name for the subtracted video. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A MgVideo pointing to the subtracted video, also stored as `subtract_video` on the parent MgVideo
    """

    of, fex = os.path.splitext(self.filename)

    if target_name is None:
        target_name = of + '_subtracted.avi'

    if not overwrite:
        target_name = generate_outfilename(target_name)

    width, height = self.width, self.height

    if bg_img is None:
        # Render an average image of the video file for background subtraction
        bg_img = musicalgestures.MgVideo(self.filename).blend(component_mode='average').filename
    else:
        # Check if background image extension is .png or not
        pass_if_container_is(".png", bg_img)

    # Set input/output and background color to white
    cmd = ['ffmpeg', '-y', '-i', bg_img, '-i', self.filename]
    cmd_end = ['-shortest', '-pix_fmt', 'yuv420p', target_name]
    cmd_filter = f'color={bg_color}:size={width}x{height} [matte];[1:0]'

    # Set color mode
    if color == True:
        pixformat = 'gbrp'
    else:
        pixformat = 'gray'

    cmd_filter += f'format={pixformat}, split[mask][video];[0:0][mask]'

    # Set frame difference
    if filtertype is not None:
        if filtertype.lower() == 'regular':
            cmd_filter += 'blend=all_mode=difference[diff],'
        else:
            cmd_filter += 'blend=all_mode=difference,'
    else:
        cmd_filter += 'blend=all_mode=difference,'

    thresh_color = matplotlib.colors.to_hex([threshold, threshold, threshold])
    thresh_color = '0x' + thresh_color[1:]

    # Set threshold
    if filtertype is not None:
        if filtertype.lower() == 'regular':
            cmd += ['-f', 'lavfi', '-i', f'color={thresh_color},scale={width}:{height}', 
                    '-f', 'lavfi', '-i', f'color=black,scale={width}:{height}']
            cmd_filter += '[1][diff]threshold,'
        elif filtertype.lower() == 'binary':
            cmd += ['-f', 'lavfi', '-i', f'color={thresh_color},scale={width}:{height}', '-f', 'lavfi', '-i',
                    f'color=black,scale={width}:{height}', '-f', 'lavfi', '-i', f'color=white,scale={width}:{height}']
            cmd_filter += ' threshold,'
        elif filtertype.lower() == 'blob':
            # cmd_filter += 'erosion,' # erosion is always 3x3 so we will hack it with a median filter with percentile=0 which will pick minimum values
            cmd_filter += f'median=radius={kernel_size}:percentile=0,'       

    # Set median
    if use_median and (filtertype is None or filtertype.lower() != 'blob'):  # makes no sense to median-filter the eroded video
        cmd_filter += f'median=radius={kernel_size},'

    # Set curves and equalisation filtering to a range of values between 0.1 and 0.9 
    new_curves = (((curves - 0) * (0.8 - 0.1)) / (1 - 0)) + 0.1
    cmd_filter += f"curves=m='0/0 {str(round(new_curves,2))}/0 {str(round(new_curves+0.1,2))}/1 1/1',"

    # Set blur
    if blur:
        cmd_filter += 'format=gray,smartblur=1,smartblur=3,'

    cmd_filter += f'format=gray [mask];[matte][video][mask] maskedmerge, format={pixformat}' 
    filter_flags = ['-filter_complex', cmd_filter]
    cmd = cmd + filter_flags + cmd_end

    ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Subtracting background:', stream=True)

    # Save subtracted video as subtract_video for parent MgVideo. NB: not
    # `self.subtract`, which would overwrite (shadow) the bound method and
    # break any subsequent subtract() call on the same object.
    self.subtract_video = musicalgestures.MgVideo(target_name, color=color, returned_by_process=True)

    return self.subtract_video

ssm

ssm(features='motiongrams', filtertype='Regular', threshold=0.05, blur='None', norm=np.inf, norm_threshold=0.001, cmap='gray_r', use_median=False, kernel_size=5, invert_yaxis=True, combine=False, title=None, target_name=None, overwrite=True)

Compute Self-Similarity Matrix (SSM) by converting the input signal into a suitable feature sequence and comparing each element of the feature sequence with all other elements of the sequence. SSMs can be computed over different input features such as 'motiongrams', 'spectrogram', 'chromagram' and 'tempogram'.

Parameters:

Name	Type	Description	Default
features	str	Defines the type of features on which to compute SSM. Possible to compute SSM on 'motiongrams', 'videograms', 'spectrogram', 'chromagram' and 'tempogram'. Defaults to 'motiongrams'.	'motiongrams'
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
norm	int	Normalize the columns of the feature sequence. Possible to compute Manhattan norm (1), Euclidean norm (2), Minimum norm (-np.inf), Maximum norm (np.inf), etc. Defaults to np.inf.	inf
norm_threshold	float	Only the columns with norm at least norm_threshold are normalized. Defaults to 0.001.	0.001
combine	bool	For 'motiongrams', compute a single SSM from the concatenated horizontal + vertical motiongram features (both axes of motion in one display) and return a single MgImage instead of an MgList of two. Defaults to False.	False
cmap	str	A Colormap instance or registered colormap name. The colormap maps the C values to colors. Defaults to 'gray_r'.	'gray_r'
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
kernel_size	int	Size of the median filter (if use_median=True) or the erosion filter (if filtertype='blob'). Defaults to 5.	5
invert_yaxis	bool	Whether to invert the y axis of the SSM. Defaults to True.	True
title	str	Optionally add title to the figure. Possible to set the filename as the title using the string 'filename'. Defaults to None.	None
target_name	[type]	Target output name for the SSM. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Type	Description
'MgList | MgImage | None'	MgList | MgImage | None: With features='motiongrams', an MgList pointing to the output SSM images (as MgImages); otherwise an MgImage of the output SSM. None when the SSM could not be computed, for instance when the file has no audio track to analyse.

Source code in musicalgestures/_ssm.py

def mg_ssm(
        self,
        features: str = 'motiongrams',
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        norm: int | float = np.inf,
        norm_threshold: float = 0.001,
        cmap: str = 'gray_r',
        use_median: bool = False,
        kernel_size: int = 5,
        invert_yaxis: bool = True,
        combine: bool = False,
        title: str | None = None,
        target_name: str | None = None,
        overwrite: bool = True) -> "MgList | MgImage | None":
    """
    Compute Self-Similarity Matrix (SSM) by converting the input signal into a suitable feature sequence and comparing each element of the feature sequence with all other elements of the sequence.
    SSMs can be computed over different input features such as 'motiongrams', 'spectrogram', 'chromagram' and 'tempogram'.

    Args:
        features (str, optional): Defines the type of features on which to compute SSM. Possible to compute SSM on 'motiongrams', 'videograms', 'spectrogram', 'chromagram' and 'tempogram'. Defaults to 'motiongrams'.
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        norm (int, optional): Normalize the columns of the feature sequence. Possible to compute Manhattan norm (1), Euclidean norm (2), Minimum norm (-np.inf), Maximum norm (np.inf), etc. Defaults to np.inf.
        norm_threshold (float, optional): Only the columns with norm at least `norm_threshold` are normalized. Defaults to 0.001.
        combine (bool, optional): For 'motiongrams', compute a single SSM from the concatenated
            horizontal + vertical motiongram features (both axes of motion in one display) and
            return a single MgImage instead of an MgList of two. Defaults to False.
        cmap (str, optional): A Colormap instance or registered colormap name. The colormap maps the C values to colors. Defaults to 'gray_r'.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        kernel_size (int, optional):  Size of the median filter (if `use_median=True`) or the erosion filter (if `filtertype='blob'`). Defaults to 5.
        invert_yaxis (bool, optional): Whether to invert the y axis of the SSM. Defaults to True.
        title (str, optional): Optionally add title to the figure. Possible to set the filename as the title using the string 'filename'. Defaults to None.
        target_name ([type], optional): Target output name for the SSM. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgList | MgImage | None: With `features='motiongrams'`, an MgList pointing to the
            output SSM images (as MgImages); otherwise an MgImage of the output SSM. None
            when the SSM could not be computed, for instance when the file has no audio
            track to analyse.
    """

    # Save figure to png
    if target_name is None:
        target_name = self.of + '_ssm.png'
    else:
        # enforce png
        target_name = os.path.splitext(target_name)[0] + '.png'
    if not overwrite:
        target_name = generate_outfilename(target_name)

    if features == 'motiongrams':
        # Make sure the file is a video file
        if self.__class__.__name__ == 'MgAudio':
            try:
                width, height = get_widthheight(self.filename)
            except:
                print(f'The "{features}" parameter works only on video files. Try "spectrogram", "chromagram" or "tempogram".')
                return None

        out_x, out_y = None, None
        target_name_mgx = os.path.splitext(target_name)[0] + '_mgv.png'
        target_name_mgy = os.path.splitext(target_name)[0] + '_mgh.png'

        if not overwrite:
            out_x = generate_outfilename(target_name_mgx)
            out_y = generate_outfilename(target_name_mgy)
        else:
            out_x, out_y = target_name_mgx, target_name_mgy

        mg_motiongrams(
            self,
            filtertype=filtertype,
            threshold=threshold,
            blur=blur,
            use_median=use_median,
            kernel_size=kernel_size,
            inverted_motiongram=False,
            equalize_motiongram=True,
            target_name_mgx=out_x,
            target_name_mgy=out_y,
            overwrite=True)

        # Normalize feature sequence
        X = librosa.util.normalize(self.ssm_figure.data[0].astype('float32'), norm=norm, threshold=norm_threshold)
        Y = librosa.util.normalize(self.ssm_figure.data[1].astype('float32'), norm=norm, threshold=norm_threshold)

        # Combined SSM: stack horizontal + vertical motiongram features per frame so a
        # single self-similarity matrix reflects both axes of motion at once.
        if combine:
            n = min(X.shape[-1], Y.shape[-1])
            XY = np.concatenate([X[..., :n], Y[..., :n]], axis=0)
            XY_ssm = slow_dot(np.transpose(XY), XY, self.length)

            fig, ax = plt.subplots(figsize=(8, 8))
            if title == 'filename':
                ax.set_title('Combined motion SSM: ' + os.path.basename(self.of + self.fex))
            elif title:
                ax.set_title(title)
            else:
                ax.set_title('Combined (horizontal + vertical) motion SSM')
            img = ax.imshow(XY_ssm, aspect='auto', cmap=cmap)
            if invert_yaxis:
                ax.invert_yaxis()
            ax.set_xlabel('Time [frames]')
            ax.set_ylabel('Time [frames]')
            cb_norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
            fig.colorbar(mpl.cm.ScalarMappable(norm=cb_norm, cmap=cmap), ax=ax, aspect=50)
            fig.tight_layout()
            plt.savefig(target_name, format='png', facecolor='white', transparent=False)
            plt.close()
            self.ssm_combined_image = MgImage(target_name)
            return MgImage(target_name)

        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)
        Y_ssm = slow_dot(np.transpose(Y), Y, self.length)

       # Plotting Self-Similarity Matrices for motiongrams
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Vertical motiongram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        if invert_yaxis:
            ax0.invert_yaxis()
        img0 = ax0.imshow(X, aspect='auto', cmap=cmap)
        fig.colorbar(img0, ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        img1 = ax1.imshow(X_ssm, aspect='auto', cmap=cmap)
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [frames]')
        ax1.set_ylabel('Time [frames]')
        # Normalize colobar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        plt.savefig(out_x, format='png', facecolor='white', transparent=False)
        plt.close()

        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Horizontal motiongram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        ax0.invert_yaxis()
        img0 = ax0.imshow(Y, aspect='auto', cmap=cmap)
        fig.colorbar(img0, ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        img1 = ax1.imshow(Y_ssm, aspect='auto', cmap=cmap)
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [frames]')
        ax1.set_ylabel('Time [frames]')
        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        plt.savefig(out_y, format='png', facecolor='white', transparent=False)
        plt.close()

        # mg_ssm also saves the motiongrams SSM as MgImages to
        # self.motiongram_vertical_image and self.motiongram_horizontal_image of the parent MgVideo
        return MgList(MgImage(out_x), MgImage(out_y))

    elif features == 'videograms':
        # Make sure the file is a video file
        if self.__class__.__name__ == 'MgAudio':
            try:
                width, height = get_widthheight(self.filename)
            except:
                print(f'The "{features}" parameter works only on video files. Try "spectrogram", "chromagram" or "tempogram".')
                return None

        out_x, out_y = None, None
        target_name_vgx = os.path.splitext(target_name)[0] + '_vgv.png'
        target_name_vgy = os.path.splitext(target_name)[0] + '_vgh.png'

        if not overwrite:
            out_x = generate_outfilename(target_name_vgx)
            out_y = generate_outfilename(target_name_vgy)
        else:
            out_x, out_y = target_name_vgx, target_name_vgy

        videograms = videograms_ffmpeg(self,
                                       target_name_x=out_x,
                                       target_name_y=out_y,
                                       overwrite=True)

        pb = MgProgressbar(total=self.length, prefix='Rendering self-similarity matrices:')

        # Load videograms and normalize them
        vgx = cv2.cvtColor(cv2.imread(videograms[0].filename), cv2.COLOR_RGB2GRAY)
        vgy = cv2.cvtColor(cv2.imread(videograms[1].filename), cv2.COLOR_RGB2GRAY)

        X = librosa.util.normalize(vgx.astype('float32'), norm=norm, threshold=norm_threshold)
        Y = librosa.util.normalize(vgy.astype('float32'), norm=norm, threshold=norm_threshold)
        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)
        Y_ssm = slow_dot(np.transpose(Y), Y, self.length)

       # Plotting Self-Similarity Matrices for motiongrams
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Vertical videogram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        ax0.invert_yaxis()
        img0 = ax0.imshow(X, aspect='auto', cmap=cmap)
        fig.colorbar(img0, ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        img1 = ax1.imshow(X_ssm, aspect='auto', cmap=cmap)
        ax1.invert_yaxis()
        ax1.set_xlabel('Time [frames]')
        ax1.set_ylabel('Time [frames]')
        # Normalize colobar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        plt.savefig(out_x, format='png', facecolor='white', transparent=False)
        plt.close()

        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Horizontal videogram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        ax0.invert_yaxis()
        img0 = ax0.imshow(Y, aspect='auto', cmap=cmap)
        fig.colorbar(img0, ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        img1 = ax1.imshow(Y_ssm, aspect='auto', cmap=cmap)
        ax1.invert_yaxis()
        ax1.set_xlabel('Time [frames]')
        ax1.set_ylabel('Time [frames]')
        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        self.ssm_figure = MgFigure(figure=None, figure_type='video.ssm', data=(vgx, vgy), layers=None, image=(target_name_vgx, target_name_vgy))

        plt.savefig(out_y, format='png', facecolor='white', transparent=False)
        plt.close()

        # mg_ssm also saves the motiongrams SSM as MgImages to
        # self.motiongram_vertical_image and self.motiongram_horizontal_image of the parent MgVideo
        return MgList(MgImage(out_x), MgImage(out_y))

    elif features == 'spectrogram':
        if not has_audio(self.filename):
            print('The video has no audio track.')
            return None

        # ffprobe for the rate and an extracted track for the samples: librosa 1.0
        # dropped the audioread fallback and can no longer read a video container.
        sr = get_samplerate(self.filename)
        x, sr = librosa.load(audio_source(self.filename), sr=sr)
        frame_length = 512
        hop_length = 128
        spectrogram = np.abs(librosa.stft(x, n_fft=frame_length, hop_length=hop_length))

        X, sr_X, formatter = smooth_downsample_feature_sequence(spectrogram, sr/hop_length)
        # Normalize columns of the feature sequence
        X = librosa.util.normalize(X.astype('float32'), norm=norm, threshold=norm_threshold)
        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)
       # Plotting SSM for spectrogram
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Spectrogram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        img0 = librosa.display.specshow(librosa.amplitude_to_db(X, ref=np.max), y_axis='linear', x_axis='time', cmap=cmap, sr=sr, hop_length=hop_length)
        fig.colorbar(img0, ax=ax0, format="%+2.f dB")
        # Format ticks
        ax0.xaxis.set_major_formatter(formatter)
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        left = - (frame_length / sr) / 2
        right = spectrogram.shape[1] * hop_length / sr + (frame_length / sr) / 2

        img1 = ax1.imshow(librosa.amplitude_to_db(X_ssm, ref=np.max), aspect='auto', cmap=cmap, extent=[left,right,right,left])
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [seconds]')
        ax1.set_ylabel('Time [seconds]')
        fig.colorbar(img1, ax=ax1, aspect=50, format="%+2.f dB")
        fig.tight_layout()

        self.ssm_figure = MgFigure(figure=fig, figure_type='audio.ssm', data=X_ssm, layers=None, image=target_name)

        plt.savefig(target_name, format='png', facecolor='white', transparent=False)
        plt.close()

        return MgImage(target_name)

    elif features == 'chromagram':
        if not has_audio(self.filename):
            print('The video has no audio track.')
            return None

        # ffprobe for the rate and an extracted track for the samples: librosa 1.0
        # dropped the audioread fallback and can no longer read a video container.
        sr = get_samplerate(self.filename)
        x, sr = librosa.load(audio_source(self.filename), sr=sr)
        frame_length = 512
        hop_length = 128
        spectrogram = np.abs(librosa.stft(x, n_fft=frame_length, hop_length=hop_length))
        chromagram = librosa.feature.chroma_stft(S=spectrogram, sr=sr, hop_length=hop_length, n_fft=frame_length)

        X, sr_X, formatter = smooth_downsample_feature_sequence(chromagram, sr/hop_length)
        # Normalize feature sequence
        X = librosa.util.normalize(X.astype('float32'), norm=norm, threshold=norm_threshold)
        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)

       # Plotting SSM for chromagram
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Chromagram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        img0 = librosa.display.specshow(X, y_axis='chroma', x_axis='time', cmap=cmap, sr=sr, hop_length=hop_length)

        # Format ticks
        ax0.xaxis.set_major_formatter(formatter)
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        left = - (frame_length / sr) / 2
        right = chromagram.shape[1] * hop_length / sr + (frame_length / sr) / 2

        img1 = ax1.imshow(X_ssm, aspect='auto', cmap=cmap, extent=[left,right,right,left])
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [seconds]')
        ax1.set_ylabel('Time [seconds]')
        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        self.ssm_figure = MgFigure(figure=fig, figure_type='audio.ssm', data=X_ssm, layers=None, image=target_name)

        plt.savefig(target_name, format='png', facecolor='white', transparent=False)
        plt.close()

        return MgImage(target_name)

    elif features == 'tempogram':
        if not has_audio(self.filename):
            print('The video has no audio track.')
            return None

        # ffprobe for the rate and an extracted track for the samples: librosa 1.0
        # dropped the audioread fallback and can no longer read a video container.
        sr = get_samplerate(self.filename)
        x, sr = librosa.load(audio_source(self.filename), sr=sr)
        frame_length = 1024
        hop_length = 512

        oenv = librosa.onset.onset_strength(y=x, sr=sr, hop_length=hop_length)
        tempogram = librosa.feature.tempogram(onset_envelope=oenv, sr=sr, hop_length=hop_length, win_length=frame_length)
        # Estimate the global tempo for display purposes
        tempo = librosa.feature.tempo(onset_envelope=oenv, sr=sr, hop_length=hop_length)[0]

        X, sr_X, formatter = smooth_downsample_feature_sequence(tempogram, sr/hop_length)
        # Normalize feature sequence
        X = librosa.util.normalize(X.astype('float32'), norm=norm, threshold=norm_threshold)
        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)

       # Plotting SSM for tempogram
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Tempogram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        img0 = librosa.display.specshow(X, y_axis='tempo', x_axis='time', cmap=cmap, sr=sr, hop_length=hop_length)

        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax0, aspect=15)
        ax0.axhline(tempo, color='w', linestyle='--', alpha=1, label='Estimated tempo={:g}'.format(tempo))
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        ax0.xaxis.set_major_formatter(formatter)
        ax0.legend(loc='upper right')
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        left = - (frame_length / sr) / 2
        right = tempogram.shape[1] * hop_length / sr + (frame_length / sr) / 2

        img1 = ax1.imshow(X_ssm, aspect='auto', cmap=cmap, extent=[left,right,right,left])
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [seconds]')
        ax1.set_ylabel('Time [seconds]')
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        self.ssm_figure = MgFigure(figure=fig, figure_type='audio.ssm', data=X_ssm, layers=None, image=target_name)

        plt.savefig(target_name, format='png', facecolor='white', transparent=False)
        plt.close()

        return MgImage(target_name)

    else:
        print(f'Unrecognized feature: "{features}". Try "motiongrams", "videograms, "spectrogram", "chromagram" or "tempogram".')
        return None

videograms

videograms(target_name_x=None, target_name_y=None, overwrite=True, mode='average', line_x=None, line_y=None)

Renders horizontal and vertical videograms of the source video using ffmpeg. By default, videoframes are averaged by axes. Alternatively, mode='slit' samples a single column and row per frame (photo-finish style) and stacks those over time.

Parameters:

Name	Type	Description	Default
target_name_x	str	Target output name for the vertical videogram (the x-axis collapse). Defaults to None (which assumes that the input filename with the suffix "_vgv" should be used).	None
target_name_y	str	Target output name for the horizontal videogram (the y-axis collapse). Defaults to None (which assumes that the input filename with the suffix "_vgh" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True
mode	str	Either 'average' (default) or 'slit'. 'slit' uses one column/row line per frame instead of averaging.	'average'
line_x	int	Source x-position (column index) used in slit mode for the vertical videogram. Defaults to the center column.	None
line_y	int	Source y-position (row index) used in slit mode for the horizontal videogram. Defaults to the center row.	None

Returns:

Name	Type	Description
MgList	'MgList'	An MgList with the MgImage objects referring to the vertical and horizontal videograms respectively.

Source code in musicalgestures/_videograms.py

def videograms_ffmpeg(
    self,
    target_name_x: str | None = None,
    target_name_y: str | None = None,
    overwrite: bool = True,
    mode: str = "average",
    line_x: int | None = None,
    line_y: int | None = None,
) -> "MgList":
    """
    Renders horizontal and vertical videograms of the source video using ffmpeg.
    By default, videoframes are averaged by axes. Alternatively, ``mode='slit'``
    samples a single column and row per frame (photo-finish style) and stacks
    those over time.

    Args:
        target_name_x (str, optional): Target output name for the vertical videogram (the x-axis collapse). Defaults to None (which assumes that the input filename with the suffix "_vgv" should be used).
        target_name_y (str, optional): Target output name for the horizontal videogram (the y-axis collapse). Defaults to None (which assumes that the input filename with the suffix "_vgh" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.
        mode (str, optional): Either ``'average'`` (default) or ``'slit'``.
            ``'slit'`` uses one column/row line per frame instead of averaging.
        line_x (int, optional): Source x-position (column index) used in slit
            mode for the vertical videogram. Defaults to the center column.
        line_y (int, optional): Source y-position (row index) used in slit mode
            for the horizontal videogram. Defaults to the center row.

    Returns:
        MgList: An MgList with the MgImage objects referring to the vertical and horizontal videograms respectively. 
    """

    width, height = get_widthheight(self.filename)
    framecount = get_framecount(self.filename)
    mode = mode.lower()

    if mode not in ["average", "slit"]:
        raise ValueError("mode must be 'average' or 'slit'.")

    def _resolve_line(line: int | None, max_size: int, name: str, default: int) -> int:
        if line is None:
            return default
        if isinstance(line, bool) or not isinstance(line, int):
            raise ValueError(f"{name} must be an integer in the range [0, {max_size - 1}].")
        if line < 0 or line >= max_size:
            raise ValueError(f"{name} must be an integer in the range [0, {max_size - 1}].")
        return line

    slit_x = width // 2
    slit_y = height // 2
    if mode == "slit":
        slit_x = _resolve_line(line_x, width, "line_x", width // 2)
        slit_y = _resolve_line(line_y, height, "line_y", height // 2)

    def calc_skipfactor(width, height, framecount):
        """
        Helper function to calculate the necessary frame-skipping to avoid integer overflow. This makes sure that we can succesfully create videograms even on many-hours-long videos as well.

        Args:
            width (int): The width of the video.
            height (int): The height of the video.
            framecount (int): The number of frames in the video.

        Returns:
            list(int, int): The necessary dilation factors to apply on the video for the horizontal and vertical videograms, respectively.
        """

        intmax = 2147483647
        skipfactor_x = int(
            math.ceil(framecount*8 / (intmax / (height+128) - 1024)))
        skipfactor_y = int(
            math.ceil(framecount / (intmax / ((width*8)+1024) - 128)))
        return skipfactor_x, skipfactor_y

    testx, testy = calc_skipfactor(width, height, framecount)

    if mode == "average":
        suffix_x, suffix_y = "_vgv.png", "_vgh.png"
    else:
        suffix_x, suffix_y = "_vgv_slit.png", "_vgh_slit.png"

    def _filters(framecount: int) -> tuple[str, str]:
        if mode == "average":
            return (
                f"scale=1:{height}:sws_flags=area,normalize,tile={framecount}x1",
                f"scale={width}:1:sws_flags=area,normalize,tile=1x{framecount}",
            )
        return (
            f"format=rgb24,crop=1:{height}:{slit_x}:0,normalize,tile={framecount}x1",
            f"format=rgb24,crop={width}:1:0:{slit_y},normalize,tile=1x{framecount}",
        )

    if testx > 1 or testy > 1:
        necessary_skipfactor = max([testx, testy])
        print(f'{os.path.basename(self.filename)} is too large to process. Applying minimal skipping necessary...')

        shortened_file = skip_frames_ffmpeg(self.filename, skip=necessary_skipfactor-1)
        skip_of = os.path.splitext(shortened_file)[0]
        framecount = get_framecount(shortened_file)
        length = get_length(shortened_file)
        vf_y, vf_x = _filters(framecount)

        target_name_x = resolve_filename(skip_of, suffix_x, target_name_x, overwrite)
        target_name_y = resolve_filename(skip_of, suffix_y, target_name_y, overwrite)

        cmd = ['ffmpeg', '-y', '-i', shortened_file, '-vf',
               vf_y, '-aspect', f'{framecount}:{height}', '-frames', '1', target_name_y]
        ffmpeg_cmd(cmd, length, stream=False, pb_prefix="Rendering horizontal videogram:")

        cmd = ['ffmpeg', '-y', '-i', shortened_file, '-vf',
               vf_x, '-aspect', f'{width}:{framecount}', '-frames', '1', target_name_x]
        ffmpeg_cmd(cmd, length, stream=False, pb_prefix="Rendering vertical videogram:")

        # save results as MgImages at self.video_gram_x and self.video_gram_y for parent MgObject
        self.videogram_x_image = MgImage(target_name_x)
        self.videogram_y_image = MgImage(target_name_y)

        # return MgList([MgImage(target_name_x), MgImage(target_name_y)])
        return MgList(self.videogram_x_image, self.videogram_y_image)


    else:
        length = get_length(self.filename)
        vf_y, vf_x = _filters(framecount)

        target_name_x = resolve_filename(self.of, suffix_x, target_name_x, overwrite)
        target_name_y = resolve_filename(self.of, suffix_y, target_name_y, overwrite)

        cmd = ['ffmpeg', '-y', '-i', self.filename, '-frames', '1', '-vf',
               vf_y, '-aspect', f'{framecount}:{height}', target_name_y]
        ffmpeg_cmd(cmd, length, stream=False, pb_prefix="Rendering horizontal videogram:")

        cmd = ['ffmpeg', '-y', '-i', self.filename, '-frames', '1', '-vf',
               vf_x, '-aspect', f'{width}:{framecount}', target_name_x]
        ffmpeg_cmd(cmd, length, stream=False, pb_prefix="Rendering vertical videogram:")

        # save results as MgImages at self.videogram_x_image and self.videogram_y_image for parent MgObject
        self.videogram_x_image = MgImage(target_name_x)
        self.videogram_y_image = MgImage(target_name_y)

        # return MgList([MgImage(target_name_x), MgImage(target_name_y)])
        return MgList(self.videogram_x_image, self.videogram_y_image)

directograms

directograms(title=None, filtertype='Adaptative', threshold=0.05, kernel_size=5, convert=True, target_name=None, overwrite=True)

Compute a directogram to factor the magnitude of motion into different angles. Each columun of the directogram is computed as the weighted histogram (HISTOGRAM_BINS) of angles for the optical flow of an input frame.

Source: Abe Davis -- Visual Rhythm and Beat (section 4.1)

Parameters:

Name	Type	Description	Default
title	str	Optionally add title to the figure. Defaults to None, which uses 'Directogram' as a title. Defaults to None.	None
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.	'Adaptative'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
kernel_size	int	Size of structuring element. Defaults to 5.	5
convert	bool	If True (default), non-AVI input is first converted to an all-intra MJPEG .avi (cached as self.as_avi) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.	True
target_name	str	Target output name for the directogram. Defaults to None (which assumes that the input filename with the suffix "_dg" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	A MgFigure object referring to the internal figure and its data.

Source code in musicalgestures/_directograms.py

def mg_directograms(self, title: str | None = None, filtertype: str = 'Adaptative', threshold: float = 0.05, kernel_size: int = 5, convert: bool = True, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Compute a directogram to factor the magnitude of motion into different angles.
    Each columun of the directogram is computed as the weighted histogram (HISTOGRAM_BINS) of angles for the optical flow of an input frame.

    Source: Abe Davis -- [Visual Rhythm and Beat](http://www.abedavis.com/files/papers/VisualRhythm_Davis18.pdf) (section 4.1)

    Args:
        title (str, optional): Optionally add title to the figure. Defaults to None, which uses 'Directogram' as a title. Defaults to None.
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        convert (bool, optional): If True (default), non-AVI input is first converted to an all-intra MJPEG `.avi` (cached as `self.as_avi`) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.
        target_name (str, optional): Target output name for the directogram. Defaults to None (which assumes that the input filename with the suffix "_dg" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgFigure: A MgFigure object referring to the internal figure and its data.
    """

    of, fex = os.path.splitext(self.filename)

    if convert and fex != '.avi':
        # first check if there already is a converted version, if not create one and register it to self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex
        filename = of + fex
    else:
        filename = self.filename

    _ensure_numba()  # JIT-compile the directogram kernels on first use

    vidcap = cv2.VideoCapture(filename)
    fps = float(vidcap.get(cv2.CAP_PROP_FPS))
    length = int(vidcap.get(cv2.CAP_PROP_FRAME_COUNT))

    pb = MgProgressbar(total=length, prefix='Rendering directogram:')

    directograms = []
    directogram_times = np.zeros((length-1,))
    ret, frame = vidcap.read()
    prev_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    i = 0

    while vidcap.isOpened():

        ret, frame = vidcap.read()

        if ret == True:
            next_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            if filtertype == 'Adaptative':
                next_frame = cv2.adaptiveThreshold(next_frame, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
            else:
                # Frame Thresholding: apply threshold filter and median filter (of `kernel_size`x`kernel_size`) to the frame.
                next_frame = filter_frame(next_frame, filtertype, threshold, kernel_size)

            # Renders a dense optical flow video of the input video file using `cv2.calcOpticalFlowFarneback()`.
            # The description of the matching parameters are taken from the cv2 documentation.
            optical_flow = cv2.calcOpticalFlowFarneback(prev_frame, next_frame, None, 0.5, 3, 15, 3, 5, 1.2, 0)
            directograms.append(directogram(optical_flow))
            directogram_times[i] = len(directograms) / fps
            prev_frame = next_frame

        else:
            pb.progress(length)
            break

        pb.progress(i)
        i += 1

    vidcap.release()

    # Create and save the figure
    fig, ax = plt.subplots(figsize=(12, 4), dpi=300)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    # add title
    if title is None:
        title = os.path.basename(f'Directogram (filter type: {filtertype})')

    fig.suptitle(title, fontsize=16)

    ax.imshow(np.array(directograms).T, extent=[directogram_times.min(), directogram_times.max(), 
    HISTOGRAM_BINS.min(), HISTOGRAM_BINS.max()], norm=colors.PowerNorm(gamma=1.0/2.0), aspect='auto')

    ax.set_ylabel('Angle [Radians]')
    ax.set_xlabel('Time [Seconds]')

    target_name = resolve_filename(of, '_dg.png', target_name, overwrite)

    plt.savefig(target_name, format='png', transparent=False)
    plt.close()

    # Create MgFigure
    data = {
        "FPS": fps,
        "path": self.of,
        "directogram times": directogram_times,
        "directogram": np.array(directograms),
    }

    mgf = MgFigure(
        figure=fig,
        figure_type='video.directogram',
        data=data,
        layers=None,
        image=target_name)

    return mgf

warp_audiomotion_beats

warp_audiomotion_beats(audio_file, speed=(0.5, 2), data=None, filtertype='Adaptative', threshold=0.05, kernel_size=5, target_name=None, overwrite=True)

Warp the motion beats of the video to align with the audio beats. Davis calls these visual beats, patterns of motion that can be shifted in time to control visual rhythm; in this toolbox's terms they are motion beats, since they are computed from the directogram rather than perceived. Visual beats are warped after computing a directogram which factors the magnitude of motion in the video into different angles.

Source: Abe Davis -- Visual Rhythm and Beat (section 5)

Parameters:

Name	Type	Description	Default
audio_file	str	Path to the audio file.	required
speed	tuple	Speed's change between the audio and motion beats which can be adjusted to slow down or speed up the visual rhythms. Defaults to (0.5,2).	(0.5, 2)
data	array_like	Computed directogram data can be added separately to avoid the directogram processing time (which can be quite long). Defaults to None.	None
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.	'Adaptative'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
kernel_size	int	Size of structuring element. Defaults to 5.	5
target_name	str	Target output name for the directogram. Defaults to None (which assumes that the input filename with the suffix "_dg" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	'musicalgestures.MgVideo'	A MgVideo as warp_audiovisual_beats for parent MgVideo

Source code in musicalgestures/_warp.py

def mg_warp_audiomotion_beats(self: "musicalgestures.MgVideo", audio_file: str, speed: tuple = (0.5, 2), data=None, filtertype: str = 'Adaptative', threshold: float = 0.05, kernel_size: int = 5, target_name: str | None = None, overwrite: bool = True) -> "musicalgestures.MgVideo":
    """
    Warp the motion beats of the video to align with the audio beats. Davis calls these visual beats, patterns of motion that can be shifted in time to control visual rhythm; in this toolbox's terms they are motion beats, since they are computed from the directogram rather than perceived.
    Visual beats are warped after computing a directogram which factors the magnitude of motion in the video into different angles.

    Source: Abe Davis -- [Visual Rhythm and Beat](http://www.abedavis.com/files/papers/VisualRhythm_Davis18.pdf) (section 5)

    Args:
        audio_file (str): Path to the audio file.
        speed (tuple, optional): Speed's change between the audio and motion beats which can be adjusted to slow down or speed up the visual rhythms. Defaults to (0.5,2).
        data (array_like, optional): Computed directogram data can be added separately to avoid the directogram processing time (which can be quite long). Defaults to None.
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        target_name (str, optional): Target output name for the directogram. Defaults to None (which assumes that the input filename with the suffix "_dg" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A MgVideo as warp_audiovisual_beats for parent MgVideo
    """

    # COMPUTE DIRECTOGRAMS ------------------------------------------------------------------------------------------------------

    if data is None:
        directogram = mg_directograms(self, title=None, filtertype=filtertype, threshold=threshold, kernel_size=kernel_size, target_name=target_name, overwrite=overwrite)
        directograms = directogram.data['directogram']
        fps = directogram.data['FPS']

    else:
        directograms = data
        vidcap = cv2.VideoCapture(self.filename)
        fps = float(vidcap.get(cv2.CAP_PROP_FPS))

    # COMPUTE AUDIO AND VISUAL BEATS --------------------------------------------------------------------------------------------

    pb = MgProgressbar(total=130, prefix='Warping audio–motion beats:')
    pb.progress(0)
    # audio_source is a no-op for a real audio file and extracts the track if a video
    # is passed instead, which librosa 1.0 can no longer open by itself.
    signal, sr = librosa.load(audio_source(audio_file), mono=True)
    pb.progress(5)

    # Compute onset and impact envelopes
    onset_envelopes = librosa.onset.onset_strength(signal, sr=sr)
    pb.progress(10)
    impact_envelopes = impact_envelope(directograms)
    pb.progress(15)

    # Compute beats with librosa
    pb.progress(20)
    audio_beats = librosa.beat.beat_track(onset_envelope=onset_envelopes,sr=sr, hop_length=512, trim=False, units='samples')
    pb.progress(25)
    visual_beats = librosa.beat.beat_track(onset_envelope=impact_envelopes,sr=fps, hop_length=1.0, trim=False, units='samples')

    # WARP AUDIO AND VISUAL BEATS -----------------------------------------------------------------------------------------------

    _ensure_numba()  # JIT-compile beats_diff on first use
    audio_differences = beats_diff(audio_beats[1], signal)
    pb.progress(30)
    visual_differences = beats_diff(visual_beats[1], np.ndarray.flatten(directograms))
    pb.progress(35)

    # Asserting if the arrays have equal shape and elements
    assert np.array_equal(audio_beats[1], np.cumsum(audio_differences[:-1]))
    assert np.array_equal(visual_beats[1], np.cumsum(visual_differences[:-1]))

    pb.progress(40)
    audio_diff_size = audio_differences.size
    audio_differences_sync = []
    visual_differences_sync = []

    # Loop through each audio and visual beat difference index
    pb.progress(45)
    audio_index, visual_index = 0, 0
    audio_diff = audio_differences[audio_index]
    visual_diff = visual_differences[visual_index]

    pb.progress(50)
    while True:

        # Convert beat differences to time
        audio_time = audio_diff / sr
        visual_time = visual_diff / fps

        speed_change = visual_time / audio_time

        # If the visual beat difference is too short, we check the next index
        if speed_change < speed[0]:
            visual_index += 1
            if visual_index == visual_differences.shape[0]:
                break
            visual_diff = visual_differences[visual_index]

        # If the audio beat difference is too short, we check the next index
        elif speed[1] < speed_change:   
            audio_index += 1
            # Iterate continuously over the audio indexes until visual indexes reach the size of the visual differences array
            audio_diff += audio_differences[audio_index % audio_diff_size]

        else:
            audio_index += 1
            visual_index += 1
            audio_differences_sync.append(audio_diff)
            visual_differences_sync.append(visual_diff)
            if visual_index == visual_differences.shape[0]:
                break
            audio_diff = audio_differences[audio_index % audio_diff_size]
            visual_diff = visual_differences[visual_index]

    pb.progress(55)
    audio_beats_sync = np.cumsum(audio_differences_sync[:-1])
    pb.progress(60)
    visual_beats_sync = np.cumsum(visual_differences_sync[:-1])

    # RENDER AUDIO-MOTION BEATS --------------------------------------------------------------------------------------------------
    pb.progress(65)
    of, fex = os.path.splitext(self.filename)

    if target_name is None:
        target_name = of + '_warped.avi'
    else:
        # enforce avi
        target_name = os.path.splitext(target_name)[0] + '.avi'
    if not overwrite:
        target_name = generate_outfilename(target_name)

    pb.progress(70)
    new_length = audio_beats_sync[-1] + 1
    extended_file_name = f'{audio_file[:-4]}_{new_length}.wav'

    pb.progress(75)
    if not os.path.isfile(extended_file_name):
        data, sample_rate = librosa.load(audio_source(audio_file), mono=True)
        old_length = data.shape[0]
        tail = new_length - old_length * (new_length // old_length)
        extended_data = np.hstack(tuple([data] * (new_length // old_length) + [data[:tail]]))
        sf.write(extended_file_name, extended_data, sample_rate)

    pb.progress(80)
    if os.path.isfile(target_name):
        os.remove(target_name)

    pb.progress(85)        
    temp_file_name = of + '_temp.avi'
    filename = of + '.avi'

    pb.progress(90)
    vidcap = cv2.VideoCapture(filename)
    ret, frame = vidcap.read()
    pb.progress(95)
    output_stream = cv2.VideoWriter(temp_file_name, cv2.VideoWriter_fourcc(*'mp4v'), fps, (frame.shape[1], frame.shape[0]))

    pb.progress(100)
    if ret == True:

        # Iterate through each output frame until the final beat is reached
        last_audio_beat, last_visual_beat = 0, 0
        input_frame_index = 0

        for audio_beat, visual_beat in zip(audio_beats_sync, visual_beats_sync):
            # Output time range
            audio_start_time = last_audio_beat / sr
            audio_end_time = audio_beat / sr
            # Input time range
            visual_start_time = last_visual_beat / fps
            visual_end_time = visual_beat / fps
            # Conversion multiplier
            multiplier = (visual_end_time - visual_start_time) / (audio_end_time - audio_start_time)

            # Iterate through every output frame in the current beat range
            output_start_index = int(np.ceil(audio_start_time * fps))
            output_end_index = int(np.floor(audio_end_time * fps))

            for output_index in range(output_start_index, output_end_index + 1):

                output_time = output_index / fps
                input_time = (output_time - audio_start_time) * multiplier + visual_start_time
                input_index = round(input_time * fps)

                while input_frame_index < input_index:
                    ret, frame = vidcap.read()
                    input_frame_index += 1
                output_stream.write(frame)

            last_audio_beat, last_visual_beat = audio_beat, visual_beat

    # Close visual stream
    pb.progress(105)
    output_stream.release()
    pb.progress(110)
    vidcap.release()

    audio_file = extended_file_name

    pb.progress(115)
    cmd = f'ffmpeg -i {temp_file_name} -i {audio_file} -c:v copy -c:a aac -strict experimental -t {visual_beats_sync[-1] / fps} {wrap_str(target_name)}'

    pb.progress(120)
    subprocess.check_call(cmd, shell=True) 
    pb.progress(125)   
    os.remove(temp_file_name)
    pb.progress(130)

    # Save the warped video on the parent MgVideo. Use a distinct attribute name (not the method
    # name) so it doesn't shadow the warp_audiovisual_beats() method on the instance.
    self.warp_video = musicalgestures.MgVideo(target_name, color=self.color, returned_by_process=True)

    return self.warp_video

blur_faces

blur_faces(mask='blur', mask_image=None, mask_scale=1.0, ellipse=True, draw_heatmap=False, neighbours=32, resolution=250, draw_scores=False, save_data=True, data_format='csv', color=(0, 0, 0), use_gpu=False, target_name=None, overwrite=True)

Automatic anonymization of faces in videos. This function works by first detecting all human faces in each video frame and then applying an anonymization filter (blurring, black rectangles or images) on each detected face region.

Credits: centerface.onnx (original) and centerface.py are based on https://github.com/Star-Clouds/centerface (revision 8c39a49), released under MIT license.

Parameters:

Name	Type	Description	Default
mask	str	Mask filter mode for face regions. 'blur' applies a strong gaussian blurring, 'rectangle' draws a solid black box, 'image' replaces the face with a custom image and 'none' does leaves the input unchanged. Defaults to 'blur'.	'blur'
mask_image	str	Anonymization image path which can be used for masking face regions. This can be activated by specifying 'image' in the mask parameter. Defaults to None.	None
mask_scale	float	Scale factor for face masks, to make sure that the masks cover the complete face. Defaults to 1.0.	1.0
ellipse	bool	Mask faces with blurred ellipses. Defaults to True.	True
draw_heatmap	bool	Draw heatmap of the detected faces using the centroid of the face mask. Defaults to False.	False
neighbours	int	Number of neighbours for smoothing the heatmap image. Defaults to 32.	32
resolution	int	Number of pixel resolution for the heatmap visualization. Defaults to 250.	250
draw_scores	bool	Draw detection faceness scores onto outputs (a score between 0 and 1 that roughly corresponds to the detector's confidence that something is a face). Defaults to False.	False
save_data	bool	Whether to save the scaled coordinates of the face mask (time (ms), x1, y1, x2, y2) for each frame to a file. Defaults to True.	True
data_format	str	Specifies format of blur_faces-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, e.g. ['csv', 'txt']. Defaults to 'csv'.	'csv'
color	tuple	Customized color of the rectangle boxes. Defaults to black (0, 0, 0).	(0, 0, 0)
use_gpu	bool	Whether to attempt GPU (CUDA) acceleration for face detection. Falls back to CPU automatically if CUDA is unavailable. Defaults to False.	False
target_name	str	Target output name. Defaults to None (which assumes that the input filename with the suffix "_blurred" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo		A MgVideo as blur_faces for parent MgVideo

Source code in musicalgestures/_blurfaces.py

def mg_blurfaces(self, 
                 mask='blur', 
                 mask_image=None, 
                 mask_scale=1.0, 
                 ellipse=True, 
                 draw_heatmap=False, 
                 neighbours=32, 
                 resolution=250, 
                 draw_scores=False, 
                 save_data=True, 
                 data_format='csv', 
                 color=(0, 0, 0), 
                 use_gpu=False,
                 target_name=None, 
                 overwrite=True):
    """
    Automatic anonymization of faces in videos. 
    This function works by first detecting all human faces in each video frame and then applying an anonymization filter 
    (blurring, black rectangles or images) on each detected face region.

    Credits: `centerface.onnx` (original) and `centerface.py` are based on https://github.com/Star-Clouds/centerface (revision 8c39a49), released under [MIT license](https://github.com/Star-Clouds/CenterFace/blob/36afed/LICENSE).

    Args:
        mask (str, optional): Mask filter mode for face regions. 'blur' applies a strong gaussian blurring, 'rectangle' draws a solid black box, 'image' replaces the face with a custom image and 'none' does leaves the input unchanged. Defaults to 'blur'.
        mask_image (str, optional): Anonymization image path which can be used for masking face regions. This can be activated by specifying 'image' in the mask parameter. Defaults to None.
        mask_scale (float, optional): Scale factor for face masks, to make sure that the masks cover the complete face. Defaults to 1.0.
        ellipse (bool, optional): Mask faces with blurred ellipses. Defaults to True.
        draw_heatmap (bool, optional): Draw heatmap of the detected faces using the centroid of the face mask. Defaults to False.
        neighbours (int, optional): Number of neighbours for smoothing the heatmap image. Defaults to 32.
        resolution (int, optional): Number of pixel resolution for the heatmap visualization. Defaults to 250.
        draw_scores (bool, optional): Draw detection faceness scores onto outputs (a score between 0 and 1 that roughly corresponds to the detector's confidence that something is a face). Defaults to False.
        save_data (bool, optional): Whether to save the scaled coordinates of the face mask (time (ms), x1, y1, x2, y2) for each frame to a file. Defaults to True.
        data_format (str, optional): Specifies format of blur_faces-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, e.g. ['csv', 'txt']. Defaults to 'csv'.
        color (tuple, optional): Customized color of the rectangle boxes. Defaults to black (0, 0, 0).
        use_gpu (bool, optional): Whether to attempt GPU (CUDA) acceleration for face detection. Falls back to CPU automatically if CUDA is unavailable. Defaults to False.
        target_name (str, optional): Target output name. Defaults to None (which assumes that the input filename with the suffix "_blurred" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A MgVideo as blur_faces for parent MgVideo
    """

    of, fex = os.path.splitext(self.filename)

    if target_name is None:
        # keep the source container (e.g. .mp4) by default
        target_name = of + '_blurred' + fex
    if not overwrite:
        target_name = generate_outfilename(target_name)
    if os.path.isfile(target_name):
        os.remove(target_name)

    pb = MgProgressbar(total=self.length, prefix='Blurring faces:')

    # Create an instance of the CenterFace class
    centerface = CenterFace(use_gpu=use_gpu)
    # Output is written through an FFmpeg pipe (libx264) for a small, portable file
    video_out = None
    # Create an empty list to append the mask coordinates
    data = []

    # Define ffmpeg command start and end
    cmd = ['ffmpeg', '-y', '-i', self.filename]
    process = ffmpeg_cmd(cmd, total_time=self.length, pipe='read')

    i = 0

    while True:
        # Read frame-by-frame
        if self.color:
            out = process.stdout.read(self.width*self.height*3)
        else:
            out = process.stdout.read(self.width*self.height)
        if out == b'':
            pb.progress(self.length)
            break

        # Transform the bytes read into a numpy array
        frame = np.frombuffer(out, dtype=np.uint8).reshape([self.height, self.width, 3]) # height, width, channels
        frame = frame.copy() # copy frame for writing it

        h, w = frame.shape[:2]
        dets, lms = centerface(frame, h, w, threshold=0.2)

        for x, det in enumerate(dets):
            boxes, score = det[:4], det[4]
            x1, y1, x2, y2 = boxes.astype(int)
            x1, y1, x2, y2 = scaling_mask(x1, y1, x2, y2, mask_scale)
            # Clip bounding boxes coordinates to valid frame region
            y1, y2 = max(0, y1), min(frame.shape[0] - 1, y2)
            x1, x2 = max(0, x1), min(frame.shape[1] - 1, x2)

            # Mask faces with rectangles
            if mask == 'rectangle':
                # Color is set to black by default but can be changed using the color parameter.
                cv2.rectangle(frame, (x1, y1), (x2, y2), color, -1)

            # Mask faces with blurred rectangles
            elif mask == 'blur':
                bf = 2  # blur factor (number of pixels in each dimension that the face will be reduced to)
                blurred_box = cv2.blur(frame[y1:y2, x1:x2], (abs(x2 - x1) // bf, abs(y2 - y1) // bf))
                # Mask faces with blurred ellipses
                if ellipse:
                    roibox = frame[y1:y2, x1:x2]
                    # Get y and x coordinate lists of the bounding ellipse
                    ey, ex = skimage.draw.ellipse((y2 - y1) // 2, (x2 - x1) // 2, (y2 - y1) // 2, (x2 - x1) // 2)
                    roibox[ey, ex] = blurred_box[ey, ex]
                    frame[y1:y2, x1:x2] = roibox
                else:
                    frame[y1:y2, x1:x2] = blurred_box

            # Mask faces with an image
            elif mask == 'image':
                target_size = (x2 - x1, y2 - y1)
                # Reading image with opencv
                mask_image = cv2.imread(mask_image, cv2.IMREAD_UNCHANGED)
                # Resizing with the target size
                resized_mask_image = cv2.resize(mask_image, target_size)
                if mask_image.shape[2] == 3:  # RGB
                    frame[y1:y2, x1:x2] = resized_mask_image
                elif mask_image.shape[2] == 4:  # RGBA
                    frame[y1:y2, x1:x2] = frame[y1:y2, x1:x2] * (1 - resized_mask_image[:, :, 3:] / 255) + resized_mask_image[:, :, :3] * (resized_mask_image[:, :, 3:] / 255)  

            # Mask nothing
            elif mask == 'none':
                pass

            # Draw heatmap of the detected faces using the centroid of the face mask.
            if draw_heatmap or save_data:
                time = frame2ms(i, self.fps)
                data.append([time, x1, y1, x2, y2])

            # Draw the faceness score (between 0 and 1) that roughly corresponds to the detector's confidence that something is a face.
            if draw_scores:
                cv2.putText(frame, f'{score:.2f}', (x1 + 0, y1 - 20), cv2.FONT_HERSHEY_DUPLEX, 0.5, (0, 255, 0))

        if video_out is None:
            cmd = ['ffmpeg', '-y', '-s', '{}x{}'.format(frame.shape[1], frame.shape[0]),
                   '-r', str(self.fps), '-f', 'rawvideo', '-pix_fmt', 'bgr24',
                   '-vcodec', 'rawvideo', '-i', '-', '-vcodec', 'libx264',
                   '-pix_fmt', 'yuv420p', target_name]
            video_out = ffmpeg_cmd(cmd, total_time=self.length, pipe='write')
        video_out.stdin.write(frame.astype(np.uint8).tobytes())

        # Flush the buffer
        process.stdout.flush()
        pb.progress(i)
        i += 1

    # Terminate the process
    process.terminate()
    if video_out is not None:
        video_out.stdin.close()
        video_out.wait()

    if self.has_audio:
        # Embed audio in the video file
        source_audio = extract_wav(of + fex)
        embed_audio_in_video(source_audio, target_name)
        os.remove(source_audio)

    # Save warped video as blur_faces for parent MgVideo
    # we have to do this here since we are not using mg_blurfaces (that would normally save the result itself)
    self.blur_faces_video = musicalgestures.MgVideo(target_name, color=self.color, returned_by_process=True)

    def save_txt(of, data, data_format, target_name=target_name, overwrite=overwrite):
        """
        Helper function to export pose estimation data as textfile(s).
        """
        def save_single_file(of, data, data_format, target_name=target_name, overwrite=overwrite):
            """
            Helper function to export pose estimation data as a textfile using pandas.
            """

            headers = ['time (ms)', 'x1', 'y1', 'x2', 'y2']
            data_format = data_format.lower()

            df = pd.DataFrame(data=data, columns=headers)

            if data_format == "tsv":
                if target_name is None:
                    target_name = of + '.tsv'
                else:
                    # take name, but enforce tsv
                    target_name = os.path.splitext(target_name)[0] + '.tsv'
                if not overwrite:
                    target_name = generate_outfilename(target_name)

                with open(target_name, 'wb') as f:
                    head_str = ''
                    for head in headers:
                        head_str += head + '\t'
                    head_str += '\n'
                    f.write(head_str.encode())
                    fmt_list = ['%.0f' for item in range(len(headers))]
                    np.savetxt(f, df.values, delimiter='\t', fmt=fmt_list)

            elif data_format == "csv":
                if target_name is None:
                    target_name = of + '.csv'
                else:
                    # take name, but enforce csv
                    target_name = os.path.splitext(target_name)[0] + '.csv'
                if not overwrite:
                    target_name = generate_outfilename(target_name)
                df.to_csv(target_name, index=None)

            elif data_format == "txt":
                if target_name is None:
                    target_name = of + '.txt'
                else:
                    # take name, but enforce txt
                    target_name = os.path.splitext(target_name)[0] + '.txt'
                if not overwrite:
                    target_name = generate_outfilename(target_name)

                with open(target_name, 'wb') as f:
                    head_str = ''
                    for head in headers:
                        head_str += head + ' '
                    head_str += '\n'
                    f.write(head_str.encode())
                    fmt_list = ['%.0f' for item in range(len(headers))]
                    np.savetxt(f, df.values, delimiter=' ', fmt=fmt_list)

            elif data_format not in ["tsv", "csv", "txt"]:
                print(f"Invalid data format: '{data_format}'.\nFalling back to '.csv'.")
                save_single_file(of, data, "csv", target_name=target_name, overwrite=overwrite)

        if type(data_format) == str:
            save_single_file(of, data, data_format, target_name=target_name, overwrite=overwrite)

        elif type(data_format) == list:
            if all([item.lower() in ["csv", "tsv", "txt"] for item in data_format]):
                data_format = list(set(data_format))
                [save_single_file(of, data, item, target_name=target_name, overwrite=overwrite)
                 for item in data_format]
            else:
                print(f"Unsupported formats in {data_format}.\nFalling back to '.csv'.")
                save_single_file(of, data, "csv", target_name=target_name, overwrite=overwrite)

    if save_data:  
        save_txt(of, data, data_format, target_name=target_name, overwrite=overwrite)  
        return self.blur_faces_video

    if draw_heatmap:
        target_name = os.path.splitext(target_name)[0] + '.png'
        if not overwrite:
            target_name = generate_outfilename(target_name)

        # Approximately update font and plot size with frame size
        plt.rcParams.update({'font.size': int((self.height/self.fps))})  
        fig, ax = plt.subplots(figsize=(int(self.width/self.fps), int(self.height/self.fps))) 
        # make sure background is white
        fig.patch.set_facecolor('white')
        fig.patch.set_alpha(1)  

        center_x, center_y = centroid_mask(np.asarray(data))
        im, extent = nearest_neighbours(center_x, center_y, self.width, self.height, resolution, neighbours)

        ax.imshow(im, extent=extent, cmap=cm.jet)
        ax.set_title(f"Heatmap of face detection (neighbours={neighbours})")
        ax.set_xlabel("Video width (pixels)")
        ax.set_ylabel("Video height (pixels)")
        ax.set_xlim(extent[0], extent[1])
        ax.set_ylim(extent[2], extent[3])

        divider = make_axes_locatable(ax)
        cax = divider.append_axes("right", size="5%", pad="3%")
        normalizer = mpl.colors.Normalize(vmin=0, vmax=1)
        fig.colorbar(mpl.cm.ScalarMappable(norm=normalizer, cmap=cm.jet), cax=cax)

        plt.tight_layout()
        plt.savefig(target_name, format='png', transparent=False)
        plt.close()

        return MgImage(target_name)

    else:
        return self.blur_faces_video

impacts

impacts(title=None, detection=True, local_mean=0.1, local_maxima=0.15, filtertype='Adaptative', threshold=0.05, kernel_size=5, convert=True, target_name=None, overwrite=True)

Compute a visual analogue of an onset envelope, aslo known as an impact envelope (Abe Davis). This is computed by summing over positive entries in the columns of the directogram. This gives an impact envelope with precisely the same form as an onset envelope. To account for large outlying spikes that sometimes happen at shot boundaries (i.e., cuts), the 99th percentile of the impact envelope values are clipped to the 98th percentile. Then, the impact envelopes are normalized by their maximum to make calculations more consistent across video resolutions. Fianlly, the local mean of the impact envelopes are calculated using a 0.1-second window, and local maxima using a 0.15-second window. Impacts are defined as local maxima that are above their local mean by at least 10% of the envelope’s global maximum.

Source: Abe Davis -- Visual Rhythm and Beat (section 4.2 and 4.3)

Parameters:

Name	Type	Description	Default
title	str	Optionally add title to the figure. Defaults to None, which uses 'Directogram' as a title. Defaults to None.	None
detection	bool	Whether to allow the detection of impacts based on local mean and local maxima or not.	True
local_mean	float	Size of the local mean window in seconds which reduces the amount of intensity variation between one impact and the next.	0.1
local_maxima	float	Size of the local maxima window in seconds for the impact envelopes	0.15
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.	'Adaptative'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
kernel_size	int	Size of structuring element. Defaults to 5.	5
convert	bool	If True (default), non-AVI input is first converted to an all-intra MJPEG .avi (cached as self.as_avi) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.	True
target_name	str	Target output name for the impacts figure. Defaults to None (which assumes that the input filename with the suffix "_impacts" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	An MgFigure object referring to the internal figure and its data.

Source code in musicalgestures/_impacts.py

def mg_impacts(self, title: str | None = None, detection: bool = True, local_mean: float = 0.1, local_maxima: float = 0.15, filtertype: str = 'Adaptative', threshold: float = 0.05, kernel_size: int = 5, convert: bool = True, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Compute a visual analogue of an onset envelope, aslo known as an impact envelope (Abe Davis).
    This is computed by summing over positive entries in the columns of the directogram. This gives an impact envelope with precisely the same
    form as an onset envelope. To account for large outlying spikes that sometimes happen at shot boundaries (i.e., cuts), the 99th percentile
    of the impact envelope values are clipped to the 98th percentile. Then, the impact envelopes are normalized by their maximum to make calculations
    more consistent across video resolutions. Fianlly, the local mean of the impact envelopes are calculated using a 0.1-second window, and local maxima
    using a 0.15-second window. Impacts are defined as local maxima that are above their local mean by at least 10% of the envelope’s global maximum.

    Source: Abe Davis -- [Visual Rhythm and Beat](http://www.abedavis.com/files/papers/VisualRhythm_Davis18.pdf) (section 4.2 and 4.3)

    Args:
        title (str, optional): Optionally add title to the figure. Defaults to None, which uses 'Directogram' as a title. Defaults to None.
        detection (bool, optional): Whether to allow the detection of impacts based on local mean and local maxima or not.
        local_mean (float, optional): Size of the local mean window in seconds which reduces the amount of intensity variation between one impact and the next.
        local_maxima (float, optional): Size of the local maxima window in seconds for the impact envelopes
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        convert (bool, optional): If True (default), non-AVI input is first converted to an all-intra MJPEG `.avi` (cached as `self.as_avi`) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.
        target_name (str, optional): Target output name for the impacts figure. Defaults to None (which assumes that the input filename with the suffix "_impacts" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgFigure: An MgFigure object referring to the internal figure and its data.
    """

    of, fex = os.path.splitext(self.filename)

    if convert and fex != '.avi':
        # first check if there already is a converted version, if not create one and register it to self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex
        filename = of + fex
    else:
        filename = self.filename

    _directograms._ensure_numba()  # JIT-compile the directogram kernels on first use
    _ensure_numba()                # JIT-compile impact_detection on first use

    vidcap = cv2.VideoCapture(filename)
    fps = float(vidcap.get(cv2.CAP_PROP_FPS))
    width = int(vidcap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(vidcap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    length = int(vidcap.get(cv2.CAP_PROP_FRAME_COUNT))

    pb = MgProgressbar(total=length, prefix='Rendering impact envelopes:')

    directograms = []
    directogram_times = []
    ret, frame = vidcap.read()
    prev_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    i = 0

    while vidcap.isOpened():

        ret, frame = vidcap.read()

        if ret == True:
            next_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            if filtertype == 'Adaptative':
                next_frame = cv2.adaptiveThreshold(
                    next_frame, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
            else:
                # Frame Thresholding: apply threshold filter and median filter (of `kernel_size`x`kernel_size`) to the frame.
                next_frame = filter_frame(next_frame, filtertype, threshold, kernel_size)

            # Renders a dense optical flow video of the input video file using `cv2.calcOpticalFlowFarneback()`.
            # The description of the matching parameters are taken from the cv2 documentation.
            optical_flow = cv2.calcOpticalFlowFarneback(
                prev_frame, next_frame, None, 0.5, 3, 15, 3, 5, 1.2, 0)
            directograms.append(_directograms.directogram(optical_flow))
            directogram_times.append(len(directograms) / fps) 
            prev_frame = next_frame

        else:
            pb.progress(length)
            break

        pb.progress(i)
        i += 1

    vidcap.release()

    # Compute impact envelopes and impact detection
    impact_envelopes = impact_envelope(np.array(directograms))
    impacts = np.array(impact_detection(impact_envelopes, np.array(directogram_times), fps, local_mean=local_mean, local_maxima=local_maxima)) / fps # convert to seconds

    fig, ax = plt.subplots(figsize=(12, 4), dpi=300)

    # make sure background is white
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    # add title
    if title is None:
        title = os.path.basename(
            f'Impact Envelopes (filter type: {filtertype})')

    fig.suptitle(title, fontsize=16)

    ax.plot(directogram_times, impact_envelopes)
    ax.set_xlabel('Time [Seconds]')
    ax.set_yticks([])
    ax.margins(x=0)

    if detection:
        ax.vlines(impacts, 0, max(impact_envelopes), colors='red', linestyles='dashed',
                  label=f'Impact Detection\nLocal mean: {local_mean}\nLocal maxima: {local_maxima}')
        ax.legend(loc='upper right')

    fig.tight_layout()

    target_name = resolve_filename(of, '_impact.png', target_name, overwrite)

    plt.savefig(target_name, format='png', transparent=False)
    plt.close()

    # create MgFigure
    data = {
        "FPS": fps,
        "path": self.of,
        "impact times": directogram_times,
        "impact envelopes": impact_envelopes,
    }

    mgf = MgFigure(
        figure=fig,
        figure_type='video.impacts',
        data=data,
        layers=None,
        image=target_name)

    return mgf

grid

grid(height=300, rows=3, columns=3, padding=0, margin=0, target_name=None, overwrite=True, return_array=False)

Generates frame strip video preview using ffmpeg.

Parameters:

Name	Type	Description	Default
height	int	Frame height, width is adjusted automatically to keep the correct aspect ratio. Defaults to 300.	300
rows	int	Number of rows of the grid. Defaults to 3.	3
columns	int	Number of columns of the grid. Defaults to 3.	3
padding	int	Padding size between the frames. Defaults to 0.	0
margin	int	Margin size for the grid. Defaults to 0.	0
target_name	[type]	Target output name for the grid image. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True
return_array	bool	Whether to return an array of not. If set to False the function writes the grid image to disk. Defaults to False.	False

Returns:

Name	Type	Description
MgImage		An MgImage object referring to the internal grid image.

Source code in musicalgestures/_grid.py

def mg_grid(self, height=300, rows=3, columns=3, padding=0, margin=0, target_name=None, overwrite=True, return_array=False):
    """
    Generates frame strip video preview using ffmpeg.

    Args:
        height (int, optional): Frame height, width is adjusted automatically to keep the correct aspect ratio. Defaults to 300.
        rows (int, optional): Number of rows of the grid. Defaults to 3.
        columns (int, optional): Number of columns of the grid. Defaults to 3.
        padding (int, optional): Padding size between the frames. Defaults to 0.
        margin (int, optional): Margin size for the grid. Defaults to 0.
        target_name ([type], optional): Target output name for the grid image. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.
        return_array (bool, optional): Whether to return an array of not. If set to False the function writes the grid image to disk. Defaults to False.

    Returns:
        MgImage: An MgImage object referring to the internal grid image.
    """

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_grid.png', target_name, overwrite)

    # Get the number of frames
    cap = cv2.VideoCapture(self.filename)
    nb_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    nth_frame = int(nb_frames / (rows*columns))

    # Define the grid specifications
    width = int((float(self.width) / self.height) * height)
    grid = rf"select=not(mod(n\,{nth_frame})),scale={width}:{height},tile={columns}x{rows}:padding={padding}:margin={margin}"

    # Declare the ffmpeg commands
    if return_array:
        cmd = ['ffmpeg', '-y', '-i', self.filename, '-frames', '1', '-q:v', '0', '-vf', grid]
        process = ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering video frame grid:', pipe='load')

        # Convert bytes to array and convert from BGR to RGB
        array = np.frombuffer(process.stdout, dtype=np.uint8).reshape([height*rows, int(width*columns), 3])[...,::-1] 

        return array
    else:
        cmd = ['ffmpeg', '-i', self.filename, '-y', '-frames', '1', '-q:v', '0', '-vf', grid, target_name]
        ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering video frame grid:')
        # Initialize the MgImage object
        img = MgImage(target_name)

        return img

actions_from_motion

actions_from_motion(envelope=None, fs=None, threshold=0.15, min_duration=0.1, min_gap=0.1)

Segment this video into actions and describe the shape of each.

With no envelope given, one is built from the body rather than from the picture: pose landmarks are extracted if they are not cached, and their quantity of motion becomes the envelope. That is what makes the result follow the person and not the camera --- a pan moves every pixel and moves no landmark relative to the others.

Parameters:

Name	Type	Description	Default
envelope		A motion envelope to segment. Defaults to None, meaning build one from pose. Pass your own to segment something else, and note that a pixel-derived envelope will read camera motion as action.	None
fs	float	Sampling rate of envelope. Defaults to the video's frame rate, which is right for any envelope with one value per frame.	None
threshold	float	Level counting as motion, as a fraction of the envelope's range. Defaults to 0.15.	0.15
min_duration	float	Shortest span kept, in seconds. Defaults to 0.1.	0.1
min_gap	float	Longest gap closed, in seconds. Defaults to 0.1.	0.1

Returns:

Name	Type	Description
list	list[Action]	The actions found, each carrying its shape in features. Also stored on the video as actions.

Source code in musicalgestures/_actions.py

def mg_actions(self: "musicalgestures.MgVideo", envelope=None, fs: float | None = None,
               threshold: float = 0.15, min_duration: float = 0.1,
               min_gap: float = 0.1) -> list[Action]:
    """Segment this video into actions and describe the shape of each.

    With no envelope given, one is built from the body rather than from the picture: pose
    landmarks are extracted if they are not cached, and their quantity of motion becomes
    the envelope. That is what makes the result follow the person and not the camera --- a
    pan moves every pixel and moves no landmark relative to the others.

    Args:
        envelope: A motion envelope to segment. Defaults to None, meaning build one from
            pose. Pass your own to segment something else, and note that a pixel-derived
            envelope will read camera motion as action.
        fs (float, optional): Sampling rate of `envelope`. Defaults to the video's frame
            rate, which is right for any envelope with one value per frame.
        threshold (float): Level counting as motion, as a fraction of the envelope's
            range. Defaults to 0.15.
        min_duration (float): Shortest span kept, in seconds. Defaults to 0.1.
        min_gap (float): Longest gap closed, in seconds. Defaults to 0.1.

    Returns:
        list: The actions found, each carrying its shape in `features`. Also stored on the
            video as `actions`.
    """
    rate = float(fs) if fs is not None else float(self.fps)
    if envelope is None:
        from musicalgestures._qom import pose_qom

        from musicalgestures._pose import pose_cache_landmarks

        cache = getattr(self, "_pose_keypoints", None)
        if not cache:
            self.pose()
            cache = getattr(self, "_pose_keypoints", None)
        if not cache:
            raise RuntimeError(
                "no pose landmarks are available, so there is no body to follow. Run "
                "pose() first, or pass an envelope of your own.")
        #: The cache stores flat normalised rows; pose_qom wants pixel
        #: trajectories, and a below-threshold landmark zeroed to the corner
        #: would read as a leap across the frame, so refusals become NaN.
        lm = pose_cache_landmarks(cache)
        xy = lm[:, :, :2] * np.array([[[float(self.width), float(self.height)]]])
        xy[lm[:, :, 2] == 0] = np.nan
        #: pose_qom returns (scalar QoM, per-frame speed envelope, fs): the
        #: quantity of motion is one number for the whole recording, and the
        #: series a segmenter can cut is the speed envelope.
        _qom, envelope, rate = pose_qom(xy, rate)
        envelope = np.asarray(envelope, dtype=float)

    actions = segment_actions(envelope, rate, threshold=threshold,
                              min_duration=min_duration, min_gap=min_gap,
                              source="pose-qom" if fs is None else "envelope")
    describe_actions(actions, envelope, rate)
    self.actions = actions
    return actions

postures_from_pose

postures_from_pose(landmarks=None, fs=None, stability=0.1, min_duration=1.0, min_gap=0.25, **pose_kwargs)

Segment this video into postures and describe the shape of each.

With no landmarks given, they are taken from a cached pose() result when there is one, and extracted fresh otherwise. The result is the static counterpart of actions_from_motion(): where that cuts the recording where the body moves, this cuts it where the body holds.

Parameters:

Name	Type	Description	Default
landmarks		(frames, 33, 3) landmark trajectories. Defaults to None, meaning use the cached pose data or extract them.	None
fs	float	Sampling rate of landmarks. Defaults to the video's frame rate.	None
stability	float	Fastest configuration change still counting as held, in torso lengths per second. Defaults to 0.1.	0.1
min_duration	float	Shortest hold kept, in seconds. Defaults to 1.0.	1.0
min_gap	float	Longest observed wobble bridged, in seconds. Defaults to 0.25.	0.25
**pose_kwargs		Forwarded to pose() if landmarks have to be computed.	{}

Returns:

Name	Type	Description
list	list[Posture]	The postures found, each carrying its configuration and shape. Also stored
	list[Posture]	on the video as postures.

Source code in musicalgestures/_postures.py

def mg_postures(self: "musicalgestures.MgVideo", landmarks=None, fs: float | None = None,
                stability: float = 0.1, min_duration: float = 1.0,
                min_gap: float = 0.25, **pose_kwargs) -> list[Posture]:
    """Segment this video into postures and describe the shape of each.

    With no landmarks given, they are taken from a cached ``pose()`` result when there is
    one, and extracted fresh otherwise. The result is the static counterpart of
    ``actions_from_motion()``: where that cuts the recording where the body moves, this
    cuts it where the body holds.

    Args:
        landmarks: ``(frames, 33, 3)`` landmark trajectories. Defaults to None, meaning
            use the cached pose data or extract them.
        fs (float, optional): Sampling rate of `landmarks`. Defaults to the video's frame
            rate.
        stability (float): Fastest configuration change still counting as held, in torso
            lengths per second. Defaults to 0.1.
        min_duration (float): Shortest hold kept, in seconds. Defaults to 1.0.
        min_gap (float): Longest observed wobble bridged, in seconds. Defaults to 0.25.
        **pose_kwargs: Forwarded to ``pose()`` if landmarks have to be computed.

    Returns:
        list: The postures found, each carrying its configuration and shape. Also stored
        on the video as `postures`.
    """
    rate = float(fs) if fs is not None else float(self.fps)
    if landmarks is None:
        from musicalgestures._pose import _ensure_pose_keypoints, pose_cache_landmarks

        cache = _ensure_pose_keypoints(self, **pose_kwargs)
        landmarks = pose_cache_landmarks(cache)
        if isinstance(cache, dict) and cache.get("fps") and fs is None:
            rate = float(cache["fps"])

    postures = segment_postures(landmarks, rate, stability=stability,
                                min_duration=min_duration, min_gap=min_gap,
                                source="pose-landmarks" if fs is None else "landmarks")
    describe_postures(postures)
    self.postures = postures
    return postures

resample

resample(fps=None, speed=None, skip=None, target_name=None, overwrite=True)

Resample the (already loaded) video and return a new MgVideo, leaving the original object untouched.

Three independent, combinable operations:

• fps: retime to a target frame rate using FFmpeg's fps filter — duration-preserving (frames are dropped/duplicated to hit the rate), e.g. 30 → 25 fps.
• speed: change playback speed by a factor (>1 faster/shorter, <1 slower/longer); the video is retimed with setpts and the audio with atempo so they stay in sync.
• skip: integer frame decimation — discard skip frames for every one kept (this also shortens/speeds up the clip), matching the loader's skip parameter.

When more than one is given they are applied in order: skip → speed/fps.

Parameters:

Name	Type	Description	Default
fps	float	Target frame rate (duration-preserving). Defaults to None.	None
speed	float	Playback-speed factor. Defaults to None.	None
skip	int	Discard skip frames for every one kept. Defaults to None.	None
target_name	str	Output name. Defaults to None (input filename + "_resampled").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	MgVideo	a new MgVideo pointing to the resampled file.

Source code in musicalgestures/_videoadjust.py

def mg_resample(self, fps=None, speed=None, skip=None, target_name=None, overwrite=True) -> "musicalgestures.MgVideo":
    """
    Resample the (already loaded) video and return a **new** MgVideo, leaving the original
    object untouched.

    Three independent, combinable operations:

    * ``fps``: retime to a target frame rate using FFmpeg's ``fps`` filter — **duration-preserving**
      (frames are dropped/duplicated to hit the rate), e.g. 30 → 25 fps.
    * ``speed``: change playback speed by a factor (>1 faster/shorter, <1 slower/longer); the video
      is retimed with ``setpts`` and the audio with ``atempo`` so they stay in sync.
    * ``skip``: integer frame decimation — discard ``skip`` frames for every one kept (this also
      shortens/speeds up the clip), matching the loader's ``skip`` parameter.

    When more than one is given they are applied in order: ``skip`` → ``speed``/``fps``.

    Args:
        fps (float, optional): Target frame rate (duration-preserving). Defaults to None.
        speed (float, optional): Playback-speed factor. Defaults to None.
        skip (int, optional): Discard ``skip`` frames for every one kept. Defaults to None.
        target_name (str, optional): Output name. Defaults to None (input filename + "_resampled").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgVideo: a new MgVideo pointing to the resampled file.
    """
    import musicalgestures

    if fps is None and speed is None and not skip:
        raise ValueError("Provide at least one of fps, speed, or skip.")
    if speed is not None and speed <= 0:
        raise ValueError("speed must be a positive factor (e.g. 2.0 = twice as fast).")
    if fps is not None and fps <= 0:
        raise ValueError("fps must be a positive number.")

    source = self.filename

    # 1) Integer frame decimation (also speeds up) — reuse the tested helper.
    if skip:
        source = skip_frames_ffmpeg(source, int(skip), overwrite=overwrite)

    # 2) Speed and/or frame-rate retime in a single FFmpeg pass.
    final = source
    if speed is not None or fps is not None:
        of, fex = os.path.splitext(source)
        if target_name is None:
            out = of + '_resampled' + fex
        else:
            out = os.path.splitext(target_name)[0] + fex
        if not overwrite:
            out = generate_outfilename(out)

        vfilters = []
        if speed is not None and speed != 1:
            vfilters.append(f'setpts=PTS/{speed:.6g}')
        if fps is not None:
            vfilters.append(f'fps={fps:.6g}')
        vf = ','.join(vfilters)

        if speed is not None and speed != 1 and has_audio(source):
            # Retime audio too so it stays in sync with the sped-up/slowed-down video.
            atempo_chain = _build_atempo_chain(speed)
            cmd = ['ffmpeg', '-y', '-i', source, '-filter_complex',
                   f'[0:v]{vf}[v];[0:a]{atempo_chain}[a]', '-map', '[v]', '-map', '[a]',
                   '-q:v', '3', '-shortest', out]
        else:
            # fps-only (or no audio): -vf retimes the video and passes audio through unchanged.
            cmd = ['ffmpeg', '-y', '-i', source, '-vf', vf, '-q:v', '3', out]

        ffmpeg_cmd(cmd, get_length(source), pb_prefix='Resampling:')
        final = out

    return musicalgestures.MgVideo(final, color=self.color, returned_by_process=True)

save_analysis

save_analysis(of, fps, aom, com, qom, motion_analysis, audio_descriptors, width, height, unit, title, target_name_plot, overwrite)

Helper function to plot the motion data using matplotlib.

Source code in musicalgestures/_motionvideo.py

def save_analysis(of, fps, aom, com, qom, motion_analysis, audio_descriptors, width, height, unit, title, target_name_plot, overwrite):
    """
    Helper function to plot the motion data using matplotlib.
    """
    plt.rc('text', usetex=False)
    fig = plt.figure(figsize=(12, 10), dpi=300)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)
    # add title
    fig.suptitle(title, fontsize=16)

    if motion_analysis.lower() == 'all':
        gs = gridspec.GridSpec(3, 2)
    elif motion_analysis.lower() == 'qom':
        gs = gridspec.GridSpec(2, 2)
    else:
        gs = gridspec.GridSpec(1, 2)

    # Audio descriptors
    if audio_descriptors:
        plt.close(fig)  # discard the figure created above; build a taller one instead
        fig = plt.figure(figsize=(12, 16), dpi=300)
        fig.patch.set_facecolor('white')
        fig.patch.set_alpha(1)
        # add title
        fig.suptitle(title, fontsize=16)
        descriptors = audio_descriptors.audio.descriptors(autoshow=False).data

        if motion_analysis.lower() == 'all':
            gs = gridspec.GridSpec(6, 2)
        elif motion_analysis.lower() == 'qom':
            gs = gridspec.GridSpec(5, 2)
        else:
            gs = gridspec.GridSpec(4, 2)

    # Centroid of motion (CoM)
    if motion_analysis.lower() == 'com':
        ax0 = fig.add_subplot(gs[0, :])
        ax0.scatter(com[:, 0]/width, com[:, 1]/height, s=2)
        ax0.set_xlim((0, 1))
        ax0.set_ylim((0, 1))
        ax0.set_xlabel('Pixels normalized')
        ax0.set_ylabel('Pixels normalized')
        ax0.set_title('Centroid of motion (CoM)')

    # Area of motion (AoM)
    if motion_analysis.lower() == 'aom':
        ax0 = fig.add_subplot(gs[0, :])
        ax0.scatter(aom[:, 0], aom[:, 1], c='C0', s=2)
        ax0.scatter(aom[:, 2], aom[:, 3], c='C0', s=2)
        ax0.set_xlim((0, 1))
        ax0.set_ylim((0, 1))
        ax0.set_xlabel('Pixels normalized')
        ax0.set_ylabel('Pixels normalized')
        ax0.set_title('Area of motion (AoM)')

    # Quantity of motion (QoM)
    def adjacent_values(vals, q1, q3):
        upper_adjacent_value = q3 + (q3 - q1) * 1.5
        upper_adjacent_value = np.clip(upper_adjacent_value, q3, vals[-1])

        lower_adjacent_value = q1 - (q3 - q1) * 1.5
        lower_adjacent_value = np.clip(lower_adjacent_value, vals[0], q1)
        return lower_adjacent_value, upper_adjacent_value

    if motion_analysis.lower() == 'qom':
        ax0 = fig.add_subplot(gs[0, :])
        ax0.set_title('Quantity of motion (QoM)')
        ax0.violinplot([qom[1:]/(max(qom[1:]))], showmeans=False, showmedians=True, showextrema=True, vert=False)

        quartile1, medians, quartile3 = np.percentile([qom[1:]/(max(qom[1:]))], [25, 50, 75], axis=1)
        whiskers = np.array([adjacent_values(sorted_array, q1, q3) for sorted_array, q1, q3 in zip([qom[1:]/(max(qom[1:]))], quartile1, quartile3)])
        whiskers_min, whiskers_max = whiskers[:, 0], whiskers[:, 1]

        inds = np.arange(1, len(medians) + 1)
        ax0.scatter(medians, inds, marker='o', color='white', s=60, zorder=3)
        ax0.hlines(inds, quartile1, quartile3, color='k', linestyle='-', lw=8)
        ax0.hlines(inds, whiskers_min, whiskers_max, color='k', linestyle='-', lw=1)
        ax0.set_xlabel('Pixels normalized')
        ax0.set_yticks([])
        ax0.set_yticklabels([])

        ax1 = fig.add_subplot(gs[1, :])
        if unit.lower() == 'seconds':
            ax1.set_xlabel('Time [seconds]')
        else:
            ax1.set_xlabel('Time [samples]')
            fps = 1
        ax1.set_ylabel('Pixels normalized')
        ax1.bar(np.arange(len(qom)-1)/fps, qom[1:]/(max(qom[1:])))

    # All motion data
    if motion_analysis.lower() == 'all':
        ax0 = fig.add_subplot(gs[0, 0])
        ax0.scatter(com[:, 0]/width, com[:, 1]/height, s=2)
        ax0.set_xlim((0, 1))
        ax0.set_ylim((0, 1))
        ax0.set_xlabel('Pixels normalized')
        ax0.set_ylabel('Pixels normalized')
        ax0.set_title('Centroid of motion (CoM)')

        ax0 = fig.add_subplot(gs[0, 1])
        ax0.scatter(aom[:, 0], aom[:, 1], c='C0', s=2)
        ax0.scatter(aom[:, 2], aom[:, 3], c='C0', s=2)
        ax0.set_xlim((0, 1))
        ax0.set_ylim((0, 1))
        ax0.set_xlabel('Pixels normalized')
        ax0.set_ylabel('Pixels normalized')
        ax0.set_title('Area of motion (AoM)')

        ax1 = fig.add_subplot(gs[1, :])
        ax1.set_title('Quantity of motion (QoM)')
        ax1.violinplot([qom[1:]/(max(qom[1:]))], showmeans=False, showmedians=True, showextrema=True, vert=False)
        quartile1, medians, quartile3 = np.percentile([qom[1:]/(max(qom[1:]))], [25, 50, 75], axis=1)
        whiskers = np.array([adjacent_values(sorted_array, q1, q3) for sorted_array, q1, q3 in zip([qom[1:]/(max(qom[1:]))], quartile1, quartile3)])
        whiskers_min, whiskers_max = whiskers[:, 0], whiskers[:, 1]
        inds = np.arange(1, len(medians) + 1)
        ax1.scatter(medians, inds, marker='o', color='white', s=60, zorder=3)
        ax1.hlines(inds, quartile1, quartile3, color='k', linestyle='-', lw=8)
        ax1.hlines(inds, whiskers_min, whiskers_max, color='k', linestyle='-', lw=1)
        ax1.set_xlabel('Pixels normalized')
        ax1.set_yticks([])
        ax1.set_yticklabels([])

        ax2 = fig.add_subplot(gs[2, :])
        if unit.lower() == 'seconds':
            ax2.set_xlabel('Time [seconds]')
        else:
            ax2.set_xlabel('Time [samples]')
            fps = 1
        ax2.set_ylabel('Pixels normalized')
        ax2.bar(np.arange(len(qom)-1)/fps, qom[1:]/(max(qom[1:])))

    # Plot audio descriptors
    if audio_descriptors:

        freq_ticks = [elem*100 for elem in range(10)]
        freq_ticks = [250]
        freq = 500
        while freq < descriptors['sr']/2:
            freq_ticks.append(freq)
            freq *= 1.5

        freq_ticks = [round(elem, -1) for elem in freq_ticks]
        freq_ticks_labels = [str(round(elem/1000, 1)) + 'k' if elem > 1000 else int(round(elem)) for elem in freq_ticks]

        times = librosa.times_like(descriptors['cent'], sr=descriptors['sr'], n_fft=2048, hop_length=descriptors['hop_size'])

        if unit.lower() == 'samples':
            times = times*descriptors['sr']

        if motion_analysis.lower() == 'all':
            ax3 = fig.add_subplot(gs[3, :], sharex=ax2)
        elif motion_analysis.lower() == 'qom':
            ax3 = fig.add_subplot(gs[2, :])
        else:
            ax3 = fig.add_subplot(gs[1, :])

        ax3.set_title('Audio descriptors')
        ax3.semilogy(times, descriptors['rms'][0], label='RMS Energy')
        ax3.legend(loc='upper right')

        if motion_analysis.lower() == 'all':
            ax4 = fig.add_subplot(gs[4, :], sharex=ax2)
        elif motion_analysis.lower() == 'qom':
            ax4 = fig.add_subplot(gs[3, :])
        else:
            ax4 = fig.add_subplot(gs[2, :])

        ax4.plot(times, descriptors['flatness'].T, label='Flatness', color='y')
        ax4.legend(loc='upper right')

        if motion_analysis.lower() == 'all':
            ax5 = fig.add_subplot(gs[5, :], sharex=ax2)
        elif motion_analysis.lower() == 'qom':
            ax5 = fig.add_subplot(gs[4, :])
        else:
            ax5 = fig.add_subplot(gs[3, :])

        ax5.set_ylabel('Frequency [Hz]')
        ax5.fill_between(times, descriptors['cent'][0] - descriptors['spec_bw'][0], descriptors['cent'][0] + descriptors['spec_bw'][0], alpha=0.5, label='Centroid +- bandwidth')
        ax5.plot(times, descriptors['cent'].T, label='Centroid', color='y')
        ax5.plot(times, descriptors['rolloff'][0], label='Roll-off frequency (0.99)')
        ax5.plot(times, descriptors['rolloff_min'][0], color='r',label='Roll-off frequency (0.01)')
        ax5.legend(loc='upper right')

        if unit.lower() == 'seconds':
            ax5.set_xlabel('Time [seconds]')
        else:
            ax5.set_xlabel('Time [samples]')

    fig.tight_layout()

    if target_name_plot is None:
        target_name_plot = of + '_motionplot.png'
    else:
        # enforce png
        target_name_plot = os.path.splitext(target_name_plot)[0] + '.png'
    if not overwrite:
        target_name_plot = generate_outfilename(target_name_plot)

    plt.savefig(target_name_plot, format='png', transparent=False)
    plt.close()

    return target_name_plot

show

show(filename=None, key=None, mode='windowed', window_width=640, window_height=480, window_title=None, **ipython_kwargs)

General method to show an image or video file either in a window, or inline in a jupyter notebook.

Parameters:

Name	Type	Description	Default
filename	str	If given, mg_show will show this file instead of what it inherits from its parent object. Defaults to None.	None
key	str	If given, mg_show will search for file names corresponding to certain processes you have previously rendered on your source. It is meant to be a shortcut, so you don't have to remember the exact name (and path) of eg. a motion video corresponding to your source in your MgVideo, but you rather just use MgVideo('path/to/vid.mp4').show(key='motion'). Accepted values are 'horizontal'/'vertical' (motiongram or videogram; aliases 'mgh'/'vgh' for horizontal, 'mgv'/'vgv' for vertical, and the legacy 'mgx'/'mgy'/'vgx'/'vgy' literal x/y files), 'ssm', 'blend', 'plot', 'motion', 'history', 'motionhistory', 'sparse', 'dense', 'pose', 'warp', 'blur', and 'subtract'. Defaults to None.	None
mode	str	Whether to show things in a separate window or inline in the jupyter notebook. Accepted values are 'windowed' and 'notebook'. Defaults to 'windowed'.	'windowed'
window_width	int	The width of the window. Defaults to 640.	640
window_height	int	The height of the window. Defaults to 480.	480
window_title	str	The title of the window. If None, the title of the window will be the file name. Defaults to None.	None
ipython_kwargs	dict	Additional arguments for IPython.display.Image or IPython.display.Video. Defaults to None.	{}

Source code in musicalgestures/_show.py

def mg_show(self, filename: str | None = None, key: str | None = None, mode: str = 'windowed', window_width: int = 640, window_height: int = 480, window_title: str | None = None, **ipython_kwargs):
    # def mg_show(self, filename=None, mode='windowed', window_width=640, window_height=480, window_title=None):
    """
    General method to show an image or video file either in a window, or inline in a jupyter notebook.

    Args:
        filename (str, optional): If given, `mg_show` will show this file instead of what it inherits from its parent object. Defaults to None.
        key (str, optional): If given, `mg_show` will search for file names corresponding to certain processes you have previously rendered on your source. It is meant to be a shortcut, so you don't have to remember the exact name (and path) of eg. a motion video corresponding to your source in your MgVideo, but you rather just use `MgVideo('path/to/vid.mp4').show(key='motion')`. Accepted values are 'horizontal'/'vertical' (motiongram or videogram; aliases 'mgh'/'vgh' for horizontal, 'mgv'/'vgv' for vertical, and the legacy 'mgx'/'mgy'/'vgx'/'vgy' literal x/y files), 'ssm', 'blend', 'plot', 'motion', 'history', 'motionhistory', 'sparse', 'dense', 'pose', 'warp', 'blur', and 'subtract'. Defaults to None.
        mode (str, optional): Whether to show things in a separate window or inline in the jupyter notebook. Accepted values are 'windowed' and 'notebook'. Defaults to 'windowed'.
        window_width (int, optional): The width of the window. Defaults to 640.
        window_height (int, optional): The height of the window. Defaults to 480.
        window_title (str, optional): The title of the window. If None, the title of the window will be the file name. Defaults to None.
        ipython_kwargs (dict, optional): Additional arguments for IPython.display.Image or IPython.display.Video. Defaults to None.
    """
    # Lazy import: keeps IPython out of `import musicalgestures` startup.
    from IPython.display import Image, display, HTML, Video

    def show(file, width=640, height=480, mode='windowed', title='Untitled', parent=None, **ipython_kwargs):
        """
        Helper function which actually does the "showing".

        Args:
            file (str): Path to the file.
            width (int, optional): The width of the window. Defaults to 640.
            height (int, optional): The height of the window. Defaults to 480.
            mode (str, optional): 'windowed' will use ffplay (in a separate window), while 'notebook' will use Image or Video from IPython.display. Defaults to 'windowed'.
            title (str, optional): The title of the window. Defaults to 'Untitled'.
            ipython_kwargs (dict, optional): Additional arguments for IPython.display.Image or IPython.display.Video. Defaults to None.
        """

        # Check's if the environment is a Google Colab document
        if musicalgestures._utils.in_colab():
            mode = 'notebook'
        elif musicalgestures._utils.in_ipynb():
            mode = 'notebook'

        if mode.lower() == 'windowed':
            # from musicalgestures._utils import wrap_str
            # cmd = f'ffplay {wrap_str(file)} -window_title {wrap_str(title)} -x {width} -y {height}'

            video_to_display = os.path.realpath(file)
            cmd = ' '.join(map(str, ['ffplay', video_to_display, '-window_title', title, '-x', width, '-y', height]))
            show_in_new_process(cmd)      

        elif mode.lower() == 'notebook':
            video_formats = ['.avi', '.mp4', '.mov', '.mkv', '.mpg', '.mpeg', '.webm', '.ogg', '.ts', '.wmv', '.3gp', '.lrv', '.insv', '.360', '.glv']
            image_formats = ['.jpg', '.png', '.jpeg', '.tiff', '.gif', '.bmp']

            of, file_extension = os.path.splitext(file)
            file_extension = file_extension.lower()

            if file_extension in video_formats:
                file_type = 'video'
            elif file_extension in image_formats:
                file_type = 'image'

            if file_type == 'image':
                display(Image(file))    
            elif file_type == 'video':
                if file_extension not in ['.mp4', '.webm', '.ogg']:
                    keys = parent.__dict__.keys()

                    if "as_mp4" not in keys:
                        print('Only mp4, webm and ogg videos are supported in notebook mode.')
                        video_to_display = musicalgestures._utils.convert_to_mp4(file)
                        # register converted video as_mp4 for parent MgVideo
                        parent.as_mp4 = musicalgestures.MgVideo(video_to_display)
                    else:
                        video_to_display = parent.as_mp4.filename
                else:
                    video_to_display = file

                # check width and height of video, if they are bigger than "appropriate", limit their dimensions
                video_width, video_height = musicalgestures._utils.get_widthheight(video_to_display)
                video_width = video_width if video_width <= width else width
                video_height = video_height if video_height <= height else height

                # if the video is at the same folder as the notebook, we need to use relative path
                # and if it is somewhere else, we need to embed it to make it work (neither absolute nor relative paths seem to work without embedding)
                cwd = os.getcwd().replace('\\', '/')
                file_dir = os.path.dirname(video_to_display).replace('\\', '/')

                def colab_display(video_to_display, video_width, video_height):
                  video_file = open(video_to_display, "r+b").read()
                  video_url = f"data:video/mp4;base64,{b64encode(video_file).decode()}"
                  return HTML(f"""<video width={video_width} height={video_height} controls><source src="{video_url}"></video>""")

                if file_dir == cwd:
                    try:
                        video_to_display = os.path.relpath(video_to_display, os.getcwd()).replace('\\', '/')
                        if musicalgestures._utils.in_colab():
                            display(colab_display(video_to_display, video_width, video_height))
                        else:
                            display(Video(video_to_display,width=video_width, height=video_height, **ipython_kwargs))
                    except ValueError:
                        video_to_display = os.path.abspath(video_to_display, os.getcwd()).replace('\\', '/')
                        if musicalgestures._utils.in_colab():
                            display(colab_display(video_to_display, video_width, video_height))
                        else:
                            display(Video(video_to_display, width=video_width, height=video_height, **ipython_kwargs))
                else:
                    try:
                        video_to_display = os.path.relpath(video_to_display, os.getcwd()).replace('\\', '/')
                        if musicalgestures._utils.in_colab():
                            display(colab_display(video_to_display, video_width, video_height))
                        else:
                            display(Video(video_to_display, width=video_width, height=video_height, **ipython_kwargs))
                    except ValueError:
                        video_to_display = os.path.abspath(video_to_display, os.getcwd()).replace('\\', '/')
                        if musicalgestures._utils.in_colab():
                            display(colab_display(video_to_display, video_width, video_height))
                        else:
                            display(Video(video_to_display, width=video_width,height=video_height, **ipython_kwargs))

        else:
            print(f'Unrecognized mode: "{mode}". Try "windowed" or "notebook".')

    if window_title is None:
        window_title = self.filename

    if filename is None:
        keys = self.__dict__.keys()
        if key is None:
            filename = self.filename
            show(file=filename, width=window_width,
                 height=window_height, mode=mode, title=window_title, parent=self, **ipython_kwargs)

        elif key.lower() in ('horizontal', 'vertical', 'mgh', 'mgv', 'vgh', 'vgv',
                             'mgx', 'mgy', 'vgx', 'vgy'):
            # Orientation keys for motiongrams/videograms. Canonical aliases: mgh/vgh
            # (horizontal), mgv/vgv (vertical); legacy mgx/mgy/vgx/vgy map to the same
            # orientations they always did, since the x-axis collapse is the vertical
            # gram and the y-axis collapse is the horizontal one.
            k = key.lower()
            horizontal_keys = ('horizontal', 'mgh', 'vgh', 'mgy', 'vgy')
            orientation = 'horizontal' if k in horizontal_keys else 'vertical'
            label = orientation.capitalize()
            kinds: tuple[str, ...]
            if k in ('mgh', 'mgv', 'mgx', 'mgy'):
                kinds = ('motiongram',)
            elif k in ('vgh', 'vgv', 'vgx', 'vgy'):
                kinds = ('videogram',)
            else:
                kinds = ('motiongram', 'videogram')
            target = None
            for kind in kinds:
                attr = f"{kind}_{orientation}_image"
                if attr in keys:
                    target = (kind, getattr(self, attr).filename)
                    break
            if target is None:
                raise FileNotFoundError(
                    f"There is no known {label.lower()} {' or '.join(kinds)} for this file. "
                    "Run motiongrams() or videograms() first.")
            kind, filename = target
            show(file=filename, width=window_width, height=window_height, mode=mode,
                 title=f'{label} {kind.capitalize()} | {filename}', parent=self, **ipython_kwargs)

        elif key.lower() == 'ssm':
            if "ssm_figure" in keys:
                filename = self.ssm_figure.image
                if len(filename) == 2:
                    show(file=filename[0], width=window_width, height=window_height, mode=mode, title=f'Vertical SSM | {filename}', parent=self, **ipython_kwargs)
                    show(file=filename[1], width=window_width, height=window_height, mode=mode, title=f'Horizontal SSM | {filename}', parent=self, **ipython_kwargs)
                else:    
                    show(file=filename, width=window_width, height=window_height, mode=mode, title=f'Self-Similarity Matrix | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known self-smilarity matrix for this file.")

        elif key.lower() == 'blend':
            if "blend_image" in keys:
                filename = self.blend_image.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Blended Image | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known blended image for this file.")

        elif key.lower() == 'plot':
            # filename = self.of + '_motion_com_qom.png'
            if "motion_plot_image" in keys:
                filename = self.motion_plot_image.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Centroid and Quantity of Motion | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known motion plot for this file.")

        elif key.lower() == 'motion':
            if "motion_video" in keys:
                filename = self.motion_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Motion Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known motion video for this file.")

        elif key.lower() == 'history':
            if "history_video" in keys:
                filename = self.history_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'History Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known history video for this file.")

        elif key.lower() == 'motionhistory':
            if "motion_video" in keys:
                motion_video_keys = self.motion_video.__dict__.keys()
                if "history_video" in motion_video_keys:
                    filename = self.motion_video.history_video.filename
                    show(file=filename, width=window_width,
                         height=window_height, mode=mode, title=f'Motion History Video | {filename}', parent=self, **ipython_kwargs)
                else:
                    raise FileNotFoundError(
                        "There is no known motion history video for this file.")
            else:
                raise FileNotFoundError(
                    "There is no known motion video for this file.")

        elif key.lower() == 'sparse':
            if "flow_sparse_video" in keys:
                filename = self.flow_sparse_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Sparse Optical Flow Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known sparse optial flow video for this file.")

        elif key.lower() == 'dense':
            if "flow_dense_video" in keys:
                filename = self.flow_dense_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Dense Optical Flow Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known dense optial flow video for this file.")

        elif key.lower() == 'pose':
            if "pose_video" in keys:
                filename = self.pose_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Pose Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known pose video for this file.")

        elif key.lower() == 'warp':
            if "warp_video" in keys:
                filename = self.warp_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Warp Audio–Motion Beats | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known warp audio–motion beats video for this file.")

        elif key.lower() == 'blur':
            if "blur_faces_video" in keys:
                filename = self.blur_faces_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Blur Faces Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known blur faces video for this file.")

        elif key.lower() == 'subtract':
            if "subtract_video" in keys:
                filename = self.subtract_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Background Subtraction Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError("There is no known subtract video for this file.")

        else:
            print("Unknown shorthand.\n",
                  "For images, try 'horizontal', 'vertical', 'ssm', 'blend' or 'plot'.\n",
                  "For videos try 'motion', 'history', 'motionhistory', 'sparse', 'dense', 'pose', 'warp', 'blur' or 'subtract'.")

    else:
        show(file=filename, width=window_width,
             height=window_height, mode=mode, title=window_title, parent=self, **ipython_kwargs)
    # show(file=filename, width=window_width, height=window_height, mode=mode, title=window_title)

    return self

info

info(type=None, autoshow=True, overwrite=True)

Returns info about video/audio/format file using ffprobe.

Parameters:

Name	Type	Description	Default
type	str	Type of information to retrieve. Possible choices are 'summary', 'audio', 'video', 'format' or 'frame'. Defaults to None (which gives info about video, audio and format). - 'summary': prints a human-readable table of key video properties (resolution, fps, frame count, duration, color mode, audio) and returns a dict. - 'audio' / 'video' / 'format': returns the matching ffprobe stream as a pandas DataFrame row. - 'frame': renders a bar chart of I/P/B frame sizes and returns a DataFrame. - None: returns a DataFrame with all ffprobe stream and format metadata.	None
autoshow	bool	Whether to show the I/P/B frames figure automatically. Defaults to True. NB: The type argument needs to be set to 'frame'.	True
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filename to avoid overwriting. Defaults to True.	True

Returns:

Type	Description
	dict or pandas.DataFrame: dict when type='summary', DataFrame otherwise.

Source code in musicalgestures/_info.py

def mg_info(self, type=None, autoshow=True, overwrite=True):
    """
    Returns info about video/audio/format file using ffprobe.

    Args:
        type (str, optional): Type of information to retrieve. Possible choices are 'summary', 'audio', 'video', 'format' or 'frame'. Defaults to None (which gives info about video, audio and format).
            - 'summary': prints a human-readable table of key video properties (resolution, fps, frame count, duration, color mode, audio) and returns a dict.
            - 'audio' / 'video' / 'format': returns the matching ffprobe stream as a pandas DataFrame row.
            - 'frame': renders a bar chart of I/P/B frame sizes and returns a DataFrame.
            - None: returns a DataFrame with all ffprobe stream and format metadata.
        autoshow (bool, optional): Whether to show the I/P/B frames figure automatically. Defaults to True. NB: The type argument needs to be set to 'frame'.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filename to avoid overwriting. Defaults to True.

    Returns:
        dict or pandas.DataFrame: dict when type='summary', DataFrame otherwise.
    """

    if type == 'summary':
        framecount = get_framecount(self.filename)

        # NB: for an MgVideo `self.length` is the frame *count*, not seconds
        # (for an MgAudio it is the duration in seconds and there is no fps).
        fps = getattr(self, 'fps', None)
        duration_secs = self.length / fps if fps else float(self.length)

        h = int(duration_secs // 3600)
        m = int((duration_secs % 3600) // 60)
        s = duration_secs % 60
        duration_str = f"{h}:{m:02d}:{s:05.2f}" if h else f"{m}:{s:05.2f}"

        filesize = os.path.getsize(self.filename)
        if filesize >= 1_000_000:
            size_str = f"{filesize / 1_000_000:.1f} MB"
        elif filesize >= 1_000:
            size_str = f"{filesize / 1_000:.1f} KB"
        else:
            size_str = f"{filesize} B"

        # Query codec/profile details from ffprobe
        v = _probe_stream(self.filename, 'v')
        a = _probe_stream(self.filename, 'a')

        video_codec = v.get('codec_name')
        video_profile = v.get('profile')
        pix_fmt = v.get('pix_fmt')
        color_space = v.get('color_space')
        color_profile = ', '.join(x for x in (pix_fmt, color_space) if x and x != 'unknown') or None

        audio_codec = a.get('codec_name')
        audio_sr = a.get('sample_rate')
        audio_br = a.get('bit_rate')
        audio_sr_str = f"{int(audio_sr):,} Hz" if audio_sr and audio_sr.isdigit() else None
        audio_br_str = f"{int(audio_br) // 1000} kbps" if audio_br and audio_br.isdigit() else None

        info_dict = {
            'filename':       os.path.basename(self.filename),
            'width':          self.width,
            'height':         self.height,
            'fps':            self.fps,
            'frames':         framecount,
            'duration':       round(duration_secs, 3),
            'color':          self.color,
            'video_codec':    video_codec,
            'video_profile':  video_profile,
            'pixel_format':   pix_fmt,
            'color_space':    color_space,
            'has_audio':      bool(self.has_audio),
            'audio_codec':    audio_codec,
            'audio_sample_rate': int(audio_sr) if audio_sr and audio_sr.isdigit() else None,
            'audio_bit_rate': int(audio_br) if audio_br and audio_br.isdigit() else None,
            'filesize':       filesize,
        }

        col = 14
        print(f"{'File:':<{col}} {os.path.basename(self.filename)}")
        print(f"{'Resolution:':<{col}} {self.width} × {self.height} px")
        print(f"{'Frames:':<{col}} {framecount}  @  {self.fps:g} fps")
        print(f"{'Duration:':<{col}} {duration_str}  ({duration_secs:.3f} s)")
        print(f"{'Color:':<{col}} {'color' if self.color else 'grayscale'}")
        codec_str = video_codec or 'unknown'
        if video_profile:
            codec_str += f" ({video_profile})"
        print(f"{'Video codec:':<{col}} {codec_str}")
        if color_profile:
            print(f"{'Color profile:':<{col}} {color_profile}")
        if self.has_audio:
            audio_str = audio_codec or 'unknown'
            extras = ', '.join(x for x in (audio_sr_str, audio_br_str) if x)
            if extras:
                audio_str += f" ({extras})"
            print(f"{'Audio:':<{col}} {audio_str}")
        else:
            print(f"{'Audio:':<{col}} no")
        print(f"{'File size:':<{col}} {size_str}")

        return info_dict

    # Get streams and format information (https://ffmpeg.org/ffprobe.html)
    cmd = ["ffprobe", "-hide_banner", "-loglevel", "quiet", "-show_streams", "-show_format", self.filename]
    if type == 'frame':
        if self.fex != '.mp4':
            # Convert video file to mp4 
            self.filename = convert_to_mp4(self.of + self.fex, overwrite=overwrite)
            self.of, self.fex = os.path.splitext(self.filename)
        cmd = ["ffprobe", "-hide_banner", "-loglevel", "quiet", "-v", "error", "-select_streams", "v:0", "-show_entries", "frame=pkt_size, pict_type", self.filename]

    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
    try:
        out, _ = process.communicate(timeout=10)
        splitted = out.split('\n')
    except subprocess.TimeoutExpired:
        process.kill()
    out, err = process.communicate()
    splitted = out.split('\n')

    frame = []

    # Retrieve information and export it in a dictionary
    if type == 'frame':
        current_frame = {}
        for line in [i for i in splitted if i not in ('[SIDE_DATA]', '[/SIDE_DATA]', '')]:
            if line == '[/FRAME]':
                frame.append(current_frame)
                current_frame = {}
            elif line != '[FRAME]':
                pair = line.split('=')
                current_frame[pair[0]] = pair[1]
            else:
                pass

        ipb_frames = {
                      'frame index': range(len(frame)),
                      'size (bytes)': [int(f['pkt_size']) for f in frame],
                      'type': [f['pict_type'] for f in frame]
                      }

        df = pd.DataFrame.from_dict(ipb_frames)

        if not autoshow:
            return df

        fig, ax = plt.subplots(figsize=(12,4), dpi=300)
        fig.patch.set_facecolor('white') # make sure background is white
        fig.patch.set_alpha(1)

        for i, (label, series) in enumerate(df.groupby('type')):
            plot_frames(series, label, index=i)

        # Get handles and labels
        handles, labels = plt.gca().get_legend_handles_labels()
        order = [1,2,0] # specify order of items in legend
        # Add legend to plot
        ax.legend([handles[idx] for idx in order],[labels[idx] for idx in order])
        ax.set_xlabel('Frame index')
        ax.set_ylabel('Size (bytes)')
        fig.tight_layout()

        # Save and close so display goes through the returned MgImage's show()
        target_png = self.of + '_frames.png'
        if not overwrite:
            target_png = generate_outfilename(target_png)
        fig.savefig(target_png, facecolor='white')
        plt.close(fig)
        return MgImage(target_png)

    else:
        for i, info in enumerate(splitted):
            if info == "[STREAM]" or info == "[SIDE_DATA]" or info == "[FORMAT]":        
                frame.append(dict())
                i +=1
            elif info == "[/STREAM]" or info == "[/SIDE_DATA]" or info == "[/FORMAT]" or info == "":
                i +=1
            else:
                try:
                    key, value = splitted[i].split('=')
                    frame[-1][key] = value
                except ValueError:
                    key = splitted[i]
                    frame[-1][key] = ''

        if len(frame) > 3: 
            # Merge video stream with side data dictionary
            frame[0] = {**frame[0], **frame[1]}
            frame.pop(1)

        # Create a pandas dataframe
        df = pd.DataFrame.from_dict(frame)

        df.insert(0, 'codec_type', df.pop('codec_type')) # move codec type column
        df.pop('index') # remove index column
        df = df[df.codec_type.notna()] # remove rows with nan values in codec_type column

        if type is not None:
            return df[df.codec_type == type]
        else:
            return df

history

history(filename=None, history_length=10, weights=1, normalize=False, norm_strength=1, norm_smooth=0, target_name=None, overwrite=True)

This function creates a video where each frame is the average of the N previous frames, where n is determined by history_length. The history frames are summed up and normalized, and added to the current frame to show the history. Uses ffmpeg.

Parameters:

Name	Type	Description	Default
filename	str	Path to the input video file. If None, the video file of the MgVideo is used. Defaults to None.	None
history_length	int	Number of frames to be saved in the history tail. Defaults to 10.	10
weights	int / float / list / str	Defines the weight or weights applied to the frames in the history tail. If given as list the first element in the list will correspond to the weight of the newest frame in the tail. If given as a str - like "3 1.2 1" - it will be automatically converted to a list - like [3, 1.2, 1]. Defaults to 1.	1
normalize	bool	If True, the history video will be normalized. This can be useful when processing motion (frame difference) videos. Defaults to False.	False
norm_strength	int / float	Defines the strength of the normalization where 1 represents full strength. Defaults to 1.	1
norm_smooth	int	Defines the number of previous frames to use for temporal smoothing. The input range of each channel is smoothed using a rolling average over the current frame and the norm_smooth previous frames. Defaults to 0.	0
target_name	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_history" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo		A new MgVideo pointing to the output video file.

Source code in musicalgestures/_history.py

def history_ffmpeg(self, filename: str | None = None, history_length: int = 10, weights: int | float | list | str = 1, normalize: bool = False, norm_strength: int | float = 1, norm_smooth: int = 0, target_name: str | None = None, overwrite: bool = True):
    """
    This function  creates a video where each frame is the average of the N previous frames, where n is determined by `history_length`. The history frames are summed up and normalized, and added to the current frame to show the history. Uses ffmpeg.

    Args:
        filename (str, optional): Path to the input video file. If None, the video file of the MgVideo is used. Defaults to None.
        history_length (int, optional): Number of frames to be saved in the history tail. Defaults to 10.
        weights (int/float/list/str, optional): Defines the weight or weights applied to the frames in the history tail. If given as list the first element in the list will correspond to the weight of the newest frame in the tail. If given as a str - like "3 1.2 1" - it will be automatically converted to a list - like [3, 1.2, 1]. Defaults to 1.
        normalize (bool, optional): If True, the history video will be normalized. This can be useful when processing motion (frame difference) videos. Defaults to False.
        norm_strength (int/float, optional): Defines the strength of the normalization where 1 represents full strength. Defaults to 1.
        norm_smooth (int, optional): Defines the number of previous frames to use for temporal smoothing. The input range of each channel is smoothed using a rolling average over the current frame and the `norm_smooth` previous frames. Defaults to 0.
        target_name (str, optional): Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_history" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A new MgVideo pointing to the output video file.
    """

    if filename is None:
        filename = self.filename

    of, fex = os.path.splitext(filename)

    if type(weights) in [int, float]:
        weights_map = np.ones(history_length)
        weights_map[-1] = weights
        str_weights = ' '.join([str(weight) for weight in weights_map])
    elif type(weights) == list:
        typecheck_list = [type(item) in [int, float] for item in weights]
        if False in typecheck_list:
            raise MgInputError(
                'Found wrong type(s) in the list of weights. Use ints and floats.')
        elif len(weights) > history_length:
            raise MgInputError(
                'history_length must be greater than or equal to the number of weights specified in weights.')
        else:
            weights_map = np.ones(history_length - len(weights))
            weights.reverse()
            weights_map = list(weights_map)
            weights_map += weights
            str_weights = ' '.join([str(weight) for weight in weights_map])
    elif type(weights) == str:
        weights_as_text = weights.split()
        try:
            weights_as_list = [float(item) for item in weights_as_text]
        except ValueError:
            raise MgInputError(
                'Found wrong type(s) in the list of weights. Use ints and floats.')
        if len(weights_as_list) > history_length:
            raise MgInputError(
                'history_length must be greater than or equal to the number of weights specified in weights.')
        else:
            weights_map = np.ones(history_length - len(weights_as_list))
            weights_as_list.reverse()
            weights_map = list(weights_map)
            weights_map += weights_as_list
            str_weights = ' '.join([str(weight) for weight in weights_map])
    else:
        raise MgInputError(
            'Wrong type used for weights. Use int, float, str, or list.')

    if type(normalize) != bool:
        raise MgInputError(
            'Wrong type used for normalize. Use only bool.')

    if normalize:
        if type(norm_strength) not in [int, float]:
            raise MgInputError(
                'Wrong type used for norm_strength. Use int or float.')
        if type(norm_smooth) != int:
            raise MgInputError(
                'Wrong type used for norm_smooth. Use only int.')

    target_name = resolve_filename(of, '_history' + fex, target_name, overwrite)

    if normalize:
        if norm_smooth != 0:
            cmd = ['ffmpeg', '-y', '-i', filename, '-filter_complex',
                   f'tmix=frames={history_length}:weights={str_weights},normalize=independence=0:strength={norm_strength}:smoothing={norm_smooth}', '-q:v', '3', '-c:a', 'copy', target_name]
        else:
            cmd = ['ffmpeg', '-y', '-i', filename, '-filter_complex',
                   f'tmix=frames={history_length}:weights={str_weights},normalize=independence=0:strength={norm_strength}', '-q:v', '3', '-c:a', 'copy', target_name]
    else:
        cmd = ['ffmpeg', '-y', '-i', filename, '-vf',
               f'tmix=frames={history_length}:weights={str_weights}', '-q:v', '3', '-c:a', 'copy', target_name]

    ffmpeg_cmd(cmd, get_length(filename), pb_prefix='Rendering history video:')

    # save the result as the history_video for parent MgVideo
    self.history_video = musicalgestures.MgVideo(
        target_name, color=self.color, returned_by_process=True)

    return self.history_video

history_cv2

history_cv2(filename=None, history_length=10, weights=1, convert=True, target_name=None, overwrite=True)

This function creates a video where each frame is the average of the N previous frames, where n is determined by history_length. The history frames are summed up and normalized, and added to the current frame to show the history. Uses cv2.

Parameters:

Name	Type	Description	Default
filename	str	Path to the input video file. If None, the video file of the MgVideo is used. Defaults to None.	None
history_length	int	Number of frames to be saved in the history tail. Defaults to 10.	10
weights	int / float / list	Defines the weight or weights applied to the frames in the history tail. If given as list the first element in the list will correspond to the weight of the newest frame in the tail. Defaults to 1.	1
convert	bool	If True (default), non-AVI input is first converted to an all-intra MJPEG .avi (cached as self.as_avi) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.	True
target_name	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_history" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo		A new MgVideo pointing to the output video file.

Source code in musicalgestures/_history.py

def history_cv2(self, filename: str | None = None, history_length: int = 10, weights: int | float | list = 1, convert: bool = True, target_name: str | None = None, overwrite: bool = True):
    """
    This function  creates a video where each frame is the average of the N previous frames, where n is determined by `history_length`. The history frames are summed up and normalized, and added to the current frame to show the history. Uses cv2.

    Args:
        filename (str, optional): Path to the input video file. If None, the video file of the MgVideo is used. Defaults to None.
        history_length (int, optional): Number of frames to be saved in the history tail. Defaults to 10.
        weights (int/float/list, optional): Defines the weight or weights applied to the frames in the history tail. If given as list the first element in the list will correspond to the weight of the newest frame in the tail. Defaults to 1.
        convert (bool, optional): If True (default), non-AVI input is first converted to an all-intra MJPEG `.avi` (cached as `self.as_avi`) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.
        target_name (str, optional): Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_history" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A new MgVideo pointing to the output video file.
    """

    if filename is None:
        filename = self.filename

    of, fex = os.path.splitext(filename)

    if convert and fex != '.avi':
        # first check if there already is a converted version, if not create one and register it to the parent self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex
        filename = of + fex

    video = cv2.VideoCapture(filename)
    ret, frame = video.read()
    fourcc = cv2.VideoWriter_fourcc(*'MJPG')

    fps = float(video.get(cv2.CAP_PROP_FPS))
    width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
    length = int(video.get(cv2.CAP_PROP_FRAME_COUNT))

    pb = MgProgressbar(total=length, prefix='Rendering history video:')

    target_name = resolve_filename(of, '_history' + fex, target_name, overwrite)

    out = cv2.VideoWriter(target_name, fourcc, fps, (width, height))

    ii = 0
    history: list = []
    weights_map: list[float] = [1 for weight in range(history_length+1)]

    if type(weights) in [int, float]:
        assert isinstance(weights, (int, float))
        offset = weights - 1
        weights_map[0] = weights
    elif type(weights) == list:
        offset = sum([weight - 1 for weight in weights])
        for ind, weight in enumerate(weights):
            if ind > history_length:
                break
            weights_map[ind] = weight

    denominator = history_length + 1 + offset

    while(video.isOpened()):
        ret, frame = video.read()
        if ret == True:
            if self.color == False:
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            frame = (np.array(frame)).astype(np.float32)

            if len(history) > 0:
                #history_total = frame/(len(history)+1)
                history_total = frame * weights_map[0] / denominator
                # history_total = frame
            else:
                history_total = frame

            for ind, newframe in enumerate(history):
                #history_total += newframe/(len(history)+1)
                history_total += newframe * weights_map[ind+1] / denominator
            # or however long history you would like
            if len(history) >= history_length:
                history.pop(0)  # pop first frame
            history.append(frame)
            # 0.5 to not overload it poor thing
            total = history_total.astype(np.uint64)

            if self.color == False:
                total = cv2.cvtColor(total.astype(
                    np.uint8), cv2.COLOR_GRAY2BGR)
                out.write(total)
            else:
                out.write(total.astype(np.uint8))

        else:
            pb.progress(length)
            break

        pb.progress(ii)
        ii += 1

    out.release()

    destination_video = target_name

    if self.has_audio:
        source_audio = extract_wav(self.of + self.fex)
        embed_audio_in_video(source_audio, destination_video)
        os.remove(source_audio)

    self.history_video = musicalgestures.MgVideo(
        destination_video, color=self.color, returned_by_process=True)

    # return musicalgestures.MgVideo(destination_video, color=self.color, returned_by_process=True)
    return self.history_video

blend

blend(filename=None, mode='all_mode', component_mode='average', target_name=None, overwrite=True)

Finds and saves a blended image of an input video file using FFmpeg. The FFmpeg tblend (time blend) filter takes two consecutive frames from one single stream, and outputs the result obtained by blending the new frame on top of the old frame.

Parameters:

Name	Type	Description	Default
filename	str	Path to the input video file. If None, the video file of the MgObject is used. Defaults to None.	None
mode	str	Set blend mode for specific pixel component or all pixel components. Accepted options are 'c0_mode', 'c1_mode', c2_mode', 'c3_mode' and 'all_mode'. Defaults to 'all_mode'.	'all_mode'
component_mode	str	Component mode of the FFmpeg tblend. Available values for component modes can be accessed here: https://ffmpeg.org/ffmpeg-filters.html#blend-1. Defaults to 'average'.	'average'
target_name	str	The name of the output video. Defaults to None (which assumes that the input filename with the component mode suffix should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	MgImage	A new MgImage pointing to the output image file.

Source code in musicalgestures/_blend.py

def mg_blend_image(self: "musicalgestures.MgVideo", filename=None, mode='all_mode', component_mode='average', target_name=None, overwrite=True) -> "MgImage":
    """
    Finds and saves a blended image of an input video file using FFmpeg. 
    The FFmpeg tblend (time blend) filter takes two consecutive frames from one single stream, and outputs the result obtained by blending the new frame on top of the old frame.

    Args:
        filename (str, optional): Path to the input video file. If None, the video file of the MgObject is used. Defaults to None.
        mode (str, optional): Set blend mode for specific pixel component or all pixel components. Accepted options are 'c0_mode', 'c1_mode', c2_mode', 'c3_mode' and 'all_mode'. Defaults to 'all_mode'. 
        component_mode (str, optional): Component mode of the FFmpeg tblend. Available values for component modes can be accessed here: https://ffmpeg.org/ffmpeg-filters.html#blend-1. Defaults to 'average'.
        target_name (str, optional): The name of the output video. Defaults to None (which assumes that the input filename with the component mode suffix should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgImage: A new MgImage pointing to the output image file.
    """

    if filename is None:
        filename = self.filename

    of, fex = os.path.splitext(filename)

    if target_name is None:
        target_name = of + f'_{component_mode}.png'
    if not overwrite:
        target_name = generate_outfilename(target_name)

    # Get the number of frames
    frames = get_framecount(filename)
    # Get the number of times all frames can be divided
    divider = int(np.ceil(np.log(frames / 2) / np.log(2)))

    # Define ffmpeg command
    cmd = ['ffmpeg', '-y', '-i', filename]

    cmd_filter = ''
    # Set average blur
    if self.blur.lower() == 'average':
        cmd_filter += 'avgblur=sizeX=10:sizeY=10,'
    # set color mode
    if self.color == True:
        pixformat = 'gbrp'
    else:
        pixformat = 'gray'
    cmd_filter += f'format={pixformat},'

    # Set frame blend every two frames
    cmd_filter += f'tblend={mode}={component_mode},framestep=2,' * divider + 'setpts=1*PTS'
    cmd_end = ['-frames:v', '1', target_name]
    cmd += ['-vf', cmd_filter] + cmd_end

    # Run the command using ffmpeg and wait for it to finish
    ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering blended image:')

    # Save result as the blended image for parent MgObject
    self.blend_image = MgImage(target_name)

    return self.blend_image

pixelarray

pixelarray(width=640, target_name=None, overwrite=True)

Creates a 'Frame-Averaged Pixel Array' of a video by reducing each frame to a single pixel and arranging all frames into a single image. This is equivalent to the bash script that scales each frame to 1x1 pixel and then tiles them into a grid.

Based on the original bash script concept: - Each frame is reduced to a single pixel (average color of the frame) - All pixel values are arranged in a grid with specified width - Height is calculated automatically based on total frames and width

Parameters:

Name	Type	Description	Default
width	int	Width of the output image in pixels (number of frame-pixels per row). Defaults to 640.	640
target_name	str	The name of the output image file. If None, uses input filename with 'framearray' suffix. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgImage		A new MgImage pointing to the output frame-averaged pixel array image file.

Source code in musicalgestures/_frameaverage.py

def mg_pixelarray(self, width=640, target_name=None, overwrite=True):
    """
    Creates a 'Frame-Averaged Pixel Array' of a video by reducing each frame to a single pixel
    and arranging all frames into a single image. This is equivalent to the bash script that
    scales each frame to 1x1 pixel and then tiles them into a grid.

    Based on the original bash script concept:
    - Each frame is reduced to a single pixel (average color of the frame)
    - All pixel values are arranged in a grid with specified width
    - Height is calculated automatically based on total frames and width

    Args:
        width (int, optional): Width of the output image in pixels (number of frame-pixels per row). 
                              Defaults to 640.
        target_name (str, optional): The name of the output image file. If None, uses input filename 
                                   with '_framearray_<width>' suffix. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically 
                                  increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgImage: A new MgImage pointing to the output frame-averaged pixel array image file.
    """

    target_name = resolve_filename(self.of, f"_pixelarray_{width}.png", target_name, overwrite)

    # Get video properties
    frames = get_framecount(self.filename)
    height = int(np.ceil(frames / width))
    video_length = get_length(self.filename)

    # Method 1: Using FFmpeg (similar to the bash script)
    # This directly replicates the bash script functionality
    cmd = [
        'ffmpeg', '-y', '-i', self.filename,
        '-vf', f'scale=1:1,tile={width}x{height}',
        '-frames:v', '1',
        target_name
    ]

    ffmpeg_cmd(cmd, video_length, pb_prefix='Creating frame-averaged pixel array:')

    # Save result as the frameaverage_image for parent MgVideo
    self.frameaverage_image = MgImage(target_name)

    return self.frameaverage_image

pixelarray_cv2

pixelarray_cv2(width=640, target_name=None, overwrite=True)

Alternative implementation using OpenCV for more control over the process. Creates a 'Frame-Averaged Pixel Array' by reading each frame, calculating its average color, and arranging these average colors in a grid.

Parameters:

Name	Type	Description	Default
width	int	Width of the output image in pixels. Defaults to 640.	640
target_name	str	The name of the output image file. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files. Defaults to True.	True

Returns:

Name	Type	Description
MgImage		A new MgImage pointing to the output frame-averaged pixel array image file.

Source code in musicalgestures/_frameaverage.py

def mg_pixelarray_cv2(self, width=640, target_name=None, overwrite=True):
    """
    Alternative implementation using OpenCV for more control over the process.
    Creates a 'Frame-Averaged Pixel Array' by reading each frame, calculating its average color,
    and arranging these average colors in a grid.

    Args:
        width (int, optional): Width of the output image in pixels. Defaults to 640.
        target_name (str, optional): The name of the output image file. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files. Defaults to True.

    Returns:
        MgImage: A new MgImage pointing to the output frame-averaged pixel array image file.
    """

    target_name = resolve_filename(self.of, f"_pixelarray_cv2_{width}.png", target_name, overwrite)

    # Open video
    cap = cv2.VideoCapture(self.filename)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

    # Calculate output dimensions
    height = int(np.ceil(total_frames / width))

    pb = MgProgressbar(total=total_frames, prefix='Creating frame-averaged pixel array (cv2):')

    # Create output array
    if self.color:
        output_array = np.zeros((height, width, 3), dtype=np.uint8)
    else:
        output_array = np.zeros((height, width), dtype=np.uint8)

    frame_count = 0

    try:
        while True:
            ret, frame = cap.read()
            if not ret:
                break

            # Convert to grayscale if needed
            if not self.color:
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
                average_color = np.mean(frame)
            else:
                # Calculate average color for each channel
                average_color = np.mean(frame, axis=(0, 1))

            # Calculate position in output grid
            row = frame_count // width
            col = frame_count % width

            # Only process if within our output bounds
            if row < height:
                output_array[row, col] = average_color.astype(np.uint8)

            frame_count += 1
            pb.progress(frame_count)

    finally:
        cap.release()

    # Save the image
    cv2.imwrite(target_name, output_array)

    # Save result as the frameaverage_cv2_image for parent MgVideo
    self.frameaverage_cv2_image = MgImage(target_name)

    return self.frameaverage_cv2_image

pixelarray_stats

pixelarray_stats(width=640, include_stats=True)

Creates a frame-averaged pixel array and optionally returns statistics about the video. This function provides additional information similar to the bash script's output.

Parameters:

Name	Type	Description	Default
width	int	Width of the output image in pixels. Defaults to 640.	640
include_stats	bool	Whether to return detailed statistics. Defaults to True.	True

Returns:

Name	Type	Description
dict		Dictionary containing the generated MgImage and optional statistics.

Source code in musicalgestures/_frameaverage.py

def mg_pixelarray_stats(self, width=640, include_stats=True):
    """
    Creates a frame-averaged pixel array and optionally returns statistics about the video.
    This function provides additional information similar to the bash script's output.

    Args:
        width (int, optional): Width of the output image in pixels. Defaults to 640.
        include_stats (bool, optional): Whether to return detailed statistics. Defaults to True.

    Returns:
        dict: Dictionary containing the generated MgImage and optional statistics.
    """

    # Get video properties for statistics (similar to bash script)
    cap = cv2.VideoCapture(self.filename)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    fps = cap.get(cv2.CAP_PROP_FPS)
    duration_seconds = total_frames / fps if fps > 0 else 0
    cap.release()

    # Calculate dimensions
    height = int(np.ceil(total_frames / width))

    # Create the frame-averaged pixel array
    result_image = mg_pixelarray(self, width=width)

    result = {
        'image': result_image,
        'filename': os.path.abspath(self.filename)
    }

    if include_stats:
        # Format duration as HH:MM:SS.ms
        hours = int(duration_seconds // 3600)
        minutes = int((duration_seconds % 3600) // 60)
        seconds = duration_seconds % 60
        duration_str = f"{hours:02d}:{minutes:02d}:{seconds:06.3f}"

        result.update({
            'duration': duration_str,
            'duration_seconds': int(duration_seconds),
            'fps': int(fps + 0.5),
            'total_frames': total_frames,
            'output_width': width,
            'output_height': height,
            'filter_description': f"scale=1:1,tile={width}x{height}"
        })

    return result

heatmap

heatmap(colormap='inferno', overlay=True, alpha=0.75, background_dim=0.4, blur=0, normalize=True, gamma=0.5, target_name=None, overwrite=True)

Renders a motion heatmap showing which parts of the video change the most.

The function accumulates the absolute pixel difference between consecutive frames over the whole video, producing a single image where bright/hot regions mark areas of frequent or large change and dark/cool regions mark areas that stay still. When overlay is True the heat is composited on top of a dimmed average frame, so the activity is shown in the spatial context of the scene.

Parameters:

Name	Type	Description	Default
colormap	str	Any matplotlib colormap name used to colour the heat (e.g. 'inferno', 'jet', 'viridis', 'hot', 'magma'). Defaults to 'inferno'.	'inferno'
overlay	bool	If True, composite the heatmap over a dimmed grayscale average frame so the motion is shown in context. If False, render the bare heatmap on a black background. Defaults to True.	True
alpha	float	Maximum opacity of the heat overlay in [0, 1]. Hotter pixels are more opaque. Only used when overlay=True. Defaults to 0.75.	0.75
background_dim	float	Brightness multiplier for the average-frame background in [0, 1]. Lower values make the heat stand out more. Only used when overlay=True. Defaults to 0.4.	0.4
blur	int	Radius of an optional Gaussian smoothing applied to the accumulated motion (0 disables). Gives a smoother, less speckled heatmap. Defaults to 0.	0
normalize	bool	If True, scale the accumulated motion so the most active pixel maps to the top of the colormap. Defaults to True.	True
gamma	float	Gamma applied to the normalised heat before colouring. Values < 1 boost faint motion so subtle activity is visible; 1.0 is linear. Defaults to 0.5.	0.5
target_name	str	The name of the output image. Defaults to None (which uses the input filename with the suffix "_heatmap.png").	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment the target filename to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	A new MgImage pointing to the output heatmap image file.

Source code in musicalgestures/_heatmap.py

def mg_heatmap(
        self: "musicalgestures.MgVideo",
        colormap: str = 'inferno',
        overlay: bool = True,
        alpha: float = 0.75,
        background_dim: float = 0.4,
        blur: int = 0,
        normalize: bool = True,
        gamma: float = 0.5,
        target_name: str | None = None,
        overwrite: bool = True) -> "MgImage":
    """
    Renders a motion heatmap showing which parts of the video change the most.

    The function accumulates the absolute pixel difference between consecutive frames
    over the whole video, producing a single image where bright/hot regions mark areas
    of frequent or large change and dark/cool regions mark areas that stay still. When
    ``overlay`` is True the heat is composited on top of a dimmed average frame, so the
    activity is shown in the spatial context of the scene.

    Args:
        colormap (str, optional): Any matplotlib colormap name used to colour the heat
            (e.g. 'inferno', 'jet', 'viridis', 'hot', 'magma'). Defaults to 'inferno'.
        overlay (bool, optional): If True, composite the heatmap over a dimmed grayscale
            average frame so the motion is shown in context. If False, render the bare
            heatmap on a black background. Defaults to True.
        alpha (float, optional): Maximum opacity of the heat overlay in [0, 1]. Hotter
            pixels are more opaque. Only used when ``overlay=True``. Defaults to 0.75.
        background_dim (float, optional): Brightness multiplier for the average-frame
            background in [0, 1]. Lower values make the heat stand out more. Only used
            when ``overlay=True``. Defaults to 0.4.
        blur (int, optional): Radius of an optional Gaussian smoothing applied to the
            accumulated motion (0 disables). Gives a smoother, less speckled heatmap.
            Defaults to 0.
        normalize (bool, optional): If True, scale the accumulated motion so the most
            active pixel maps to the top of the colormap. Defaults to True.
        gamma (float, optional): Gamma applied to the normalised heat before colouring.
            Values < 1 boost faint motion so subtle activity is visible; 1.0 is linear.
            Defaults to 0.5.
        target_name (str, optional): The name of the output image. Defaults to None
            (which uses the input filename with the suffix "_heatmap.png").
        overwrite (bool, optional): Whether to allow overwriting existing files or to
            automatically increment the target filename to avoid overwriting.
            Defaults to True.

    Returns:
        MgImage: A new MgImage pointing to the output heatmap image file.
    """

    target_name = resolve_filename(self.of, '_heatmap.png', target_name, overwrite)

    cap = cv2.VideoCapture(self.filename)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    accum = np.zeros((height, width), dtype=np.float64)      # accumulated motion
    bg_accum = np.zeros((height, width, 3), dtype=np.float64)  # for the average frame (RGB)
    prev_gray = None
    n = 0

    pb = MgProgressbar(total=total_frames, prefix='Rendering motion heatmap:')

    try:
        while True:
            ret, frame = cap.read()
            if not ret:
                break
            bg_accum += frame[..., ::-1].astype(np.float64)  # BGR -> RGB
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
            if prev_gray is not None:
                accum += np.abs(gray - prev_gray)
            prev_gray = gray
            n += 1
            pb.progress(n)
    finally:
        cap.release()

    pb.progress(total_frames)

    if n == 0:
        raise RuntimeError(f"Could not read any frames from {self.filename}.")

    avg_frame = np.rint(bg_accum / n).astype(np.uint8)

    # Optional smoothing of the accumulated motion
    if blur and blur > 0:
        k = int(blur) * 2 + 1
        accum = cv2.GaussianBlur(accum, (k, k), 0)

    # Normalise to [0, 1]
    if normalize and accum.max() > 0:
        heat = accum / accum.max()
    else:
        heat = np.clip(accum, 0, 1)

    # Gamma boost so subtle motion stays visible
    if gamma and gamma != 1.0:
        heat = np.power(heat, gamma)

    # Colour-map the heat (RGB, uint8)
    cmap = matplotlib.colormaps[colormap]
    heat_rgb = (cmap(heat)[..., :3] * 255).astype(np.float64)

    if overlay:
        # Dimmed grayscale average frame as background
        gray_bg = cv2.cvtColor(avg_frame, cv2.COLOR_RGB2GRAY)
        gray_bg3 = np.stack([gray_bg] * 3, axis=-1).astype(np.float64) * background_dim
        # Hotter pixels are more opaque
        a = (heat[..., None] * alpha)
        out = gray_bg3 * (1 - a) + heat_rgb * a
    else:
        out = heat_rgb

    out = np.clip(out, 0, 255).astype(np.uint8)

    # cv2 writes BGR
    cv2.imwrite(target_name, out[..., ::-1])

    # NB: store under a non-shadowing attribute so the heatmap() method stays callable
    self.heatmap_image = MgImage(target_name)
    return self.heatmap_image

motiontempo

motiontempo(fmin=0.2, fmax=8.0, dpi=300, autoshow=True, title=None, target_name=None, overwrite=True)

Estimates the dominant motion tempo of a video from its quantity of motion.

A quantity-of-motion (QoM) signal is computed as the mean absolute difference between consecutive frames. Its dominant periodicity within [fmin, fmax] is found with an FFT and reported both in Hz and in beats per minute (BPM), giving a simple estimate of the overall motion tempo (e.g. step rate of a dancer).

Parameters:

Name	Type	Description	Default
fmin	float	Lowest motion frequency to consider (Hz). Defaults to 0.2.	0.2
fmax	float	Highest motion frequency to consider (Hz). Defaults to 8.0.	8.0
dpi	int	Image quality of the rendered figure in DPI. Defaults to 300.	300
autoshow	bool	Whether to show the resulting figure automatically. Defaults to True.	True
title	str	Optionally add a title to the figure. Use 'filename' for the file name. Defaults to None.	None
target_name	str	The name of the output image. Defaults to None (which uses the input filename with the suffix "_motiontempo.png").	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment the target filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	MgFigure	An MgFigure object. Numeric results are available in .data: 'tempo_bpm', 'dominant_frequency', 'qom', 'times', 'freqs', 'spectrum', 'fps'.

Source code in musicalgestures/_motiontempo.py

def mg_motiontempo(self, fmin=0.2, fmax=8.0, dpi=300, autoshow=True, title=None, target_name=None, overwrite=True) -> "MgFigure":
    """
    Estimates the dominant motion tempo of a video from its quantity of motion.

    A quantity-of-motion (QoM) signal is computed as the mean absolute difference
    between consecutive frames. Its dominant periodicity within ``[fmin, fmax]`` is
    found with an FFT and reported both in Hz and in beats per minute (BPM), giving a
    simple estimate of the overall motion tempo (e.g. step rate of a dancer).

    Args:
        fmin (float, optional): Lowest motion frequency to consider (Hz). Defaults to 0.2.
        fmax (float, optional): Highest motion frequency to consider (Hz). Defaults to 8.0.
        dpi (int, optional): Image quality of the rendered figure in DPI. Defaults to 300.
        autoshow (bool, optional): Whether to show the resulting figure automatically. Defaults to True.
        title (str, optional): Optionally add a title to the figure. Use 'filename' for the file name. Defaults to None.
        target_name (str, optional): The name of the output image. Defaults to None
            (which uses the input filename with the suffix "_motiontempo.png").
        overwrite (bool, optional): Whether to allow overwriting existing files or to
            automatically increment the target filename. Defaults to True.

    Returns:
        MgFigure: An MgFigure object. Numeric results are available in ``.data``:
            'tempo_bpm', 'dominant_frequency', 'qom', 'times', 'freqs', 'spectrum', 'fps'.
    """
    from musicalgestures._analysis import dominant_frequency

    target_name = resolve_filename(self.of, '_motiontempo.png', target_name, overwrite)

    cap = cv2.VideoCapture(self.filename)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    fps = cap.get(cv2.CAP_PROP_FPS) or self.fps

    qom = []
    prev_gray = None
    pb = MgProgressbar(total=total_frames, prefix='Computing motion tempo:')
    n = 0
    try:
        while True:
            ret, frame = cap.read()
            if not ret:
                break
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
            if prev_gray is not None:
                qom.append(float(np.abs(gray - prev_gray).mean()))
            prev_gray = gray
            n += 1
            pb.progress(n)
    finally:
        cap.release()
    pb.progress(total_frames)

    qom = np.asarray(qom, dtype=float)
    if len(qom) < 4:
        raise RuntimeError(f"Not enough frames in {self.filename} to estimate motion tempo.")

    # Normalise QoM to [0, 1] by pixel value range (mean abs frame difference / 255)
    qom = qom / 255.0
    mean_qom = float(qom.mean())

    times = np.arange(len(qom)) / fps

    # Dominant motion frequency and its spectrum within [fmin, fmax]
    dom_freq = dominant_frequency(qom, fps, fmin=fmin, fmax=fmax)
    tempo_bpm = dom_freq * 60.0

    freqs = np.fft.rfftfreq(len(qom), d=1.0 / fps)
    spectrum = np.abs(np.fft.rfft(qom - qom.mean()))

    fig, ax = plt.subplots(nrows=2, figsize=(12, 6), dpi=dpi)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    if title is None:
        title = ''
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title, fontsize=16)

    ax[0].plot(times, qom, color='#1f77b4', lw=0.8)
    # Average quantity of motion line + number
    ax[0].axhline(mean_qom, color='#d62728', ls='--', lw=1.2,
                  label=f'avg QoM = {mean_qom:.3f}')
    ax[0].legend(loc='upper right')
    ax[0].set(title='Quantity of motion', xlabel='Time (s)', ylabel='QoM (normalised 0–1)')
    ax[0].set_xlim(0, times[-1] if len(times) else 1)
    ax[0].set_ylim(0, max(qom.max() * 1.05, 1e-6))

    mask = (freqs >= fmin) & (freqs <= fmax)
    ax[1].plot(freqs[mask], spectrum[mask], color='#ff7f0e', lw=0.9)
    # Dominant frequency line + number
    if dom_freq > 0:
        ax[1].axvline(dom_freq, color='r', ls='--', lw=1.2,
                      label=f'dominant frequency = {dom_freq:.2f} Hz ({tempo_bpm:.1f} BPM)')
        ax[1].legend(loc='upper right')
    ax[1].set(title='Motion spectrum', xlabel='Frequency (Hz)', ylabel='Magnitude')
    ax[1].set_xlim(fmin, fmax)

    plt.tight_layout(rect=[0, 0, 1, 0.96])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    data = {
        'tempo_bpm': tempo_bpm,
        'dominant_frequency': dom_freq,
        'mean_qom': mean_qom,
        'qom': qom,
        'times': times,
        'freqs': freqs,
        'spectrum': spectrum,
        'fps': fps,
    }

    mgf = MgFigure(
        figure=fig,
        figure_type='video.motiontempo',
        data=data,
        layers=None,
        image=target_name)

    return mgf

motionvectors

motionvectors(target_name=None, overwrite=True)

Renders a video visualising the motion vectors encoded in the input video.

Inter-frame codecs (MPEG-1/2/4, H.264, H.265, …) store motion vectors that describe how macroblocks move between frames. This method uses FFmpeg's codecview filter (with -flags2 +export_mvs) to draw those vectors as arrows on top of the video, giving a quick, decoder-level view of motion without any re-computation.

NB: Only codecs that actually carry motion vectors will show arrows. Intra-only formats (e.g. MJPEG, common in .avi files) have none — convert to an inter-frame codec first (e.g. via show(mode='notebook') which makes an mp4, or any mp4/h264 source) to see motion vectors.

Parameters:

Name	Type	Description	Default
target_name	str	Target output name for the video. Defaults to None (which uses the input filename with the suffix "_motionvectors").	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment the target filename. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	MgVideo	An MgVideo pointing to the rendered motion-vector video.

Source code in musicalgestures/_motionvectors.py

def mg_motionvectors(self: "musicalgestures.MgVideo", target_name=None, overwrite=True) -> "musicalgestures.MgVideo":
    """
    Renders a video visualising the motion vectors encoded in the input video.

    Inter-frame codecs (MPEG-1/2/4, H.264, H.265, …) store motion vectors that describe
    how macroblocks move between frames. This method uses FFmpeg's ``codecview`` filter
    (with ``-flags2 +export_mvs``) to draw those vectors as arrows on top of the video,
    giving a quick, decoder-level view of motion without any re-computation.

    NB: Only codecs that actually carry motion vectors will show arrows. Intra-only
    formats (e.g. MJPEG, common in ``.avi`` files) have none — convert to an inter-frame
    codec first (e.g. via ``show(mode='notebook')`` which makes an mp4, or any mp4/h264
    source) to see motion vectors.

    Args:
        target_name (str, optional): Target output name for the video. Defaults to None
            (which uses the input filename with the suffix "_motionvectors").
        overwrite (bool, optional): Whether to allow overwriting existing files or to
            automatically increment the target filename. Defaults to True.

    Returns:
        MgVideo: An MgVideo pointing to the rendered motion-vector video.
    """
    of, fex = os.path.splitext(self.filename)

    target_name = resolve_filename(of, '_motionvectors' + fex, target_name, overwrite)

    # -flags2 +export_mvs must precede -i so the decoder exports motion vectors;
    # codecview then draws them (pf=P-frame forward, bf/bb=B-frame forward/backward).
    cmd = [
        'ffmpeg', '-y', '-flags2', '+export_mvs', '-i', self.filename,
        '-vf', 'codecview=mv=pf+bf+bb', '-q:v', '3', target_name,
    ]
    ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering motion vectors:')

    self.motionvectors_video = musicalgestures.MgVideo(
        target_name, color=self.color, returned_by_process=True)
    return self.motionvectors_video

motionvectordata

motionvectordata()

Read the motion vectors the codec already computed, as numbers rather than arrows.

motionvectors() renders these as a video to look at. This returns them as data, and the difference in cost is the point: an inter-frame codec has already searched for the displacement of every macroblock, so reading its answer is close to free, while differencing pixels does the same search again in Python. On a 103-minute recording, decoding took 2.2 s and decoding with the vectors took 2.7 s, against 27 minutes for the frame-differenced quantity of motion.

What it is not. The vectors are the encoder's decisions, not a measurement of the scene: an encoder is entitled to any vector that predicts the block cheaply, and on flat or still regions it will pick one that has nothing to do with motion. The result tracks quantity of motion well where there is motion to track and poorly where there is not --- on that same corpus, median r = 0.86 in windows with motion and 0.57 in windows without. Re-encoded proxies carry their proxy encoder's vectors, not the camera's.

Vectors are normalised to point forward in time. A B-frame may predict a block from a later frame, which reverses the sign; dividing by the vector's source corrects that, so a block moving right reads as positive whichever way it was predicted. Without it a B-frame's vectors point backwards half the time and averaging them gives roughly nothing.

The reference distance is corrected from the cadence, with two residuals. source records only the direction, plus or minus one, never how many frames away the reference was, so the distance is recovered instead from the picture-type cadence: past-referencing vectors are divided by the display-order gap to the previous reference frame, which makes a P-frame after a run of B-frames read per-frame displacement rather than the whole span's. What the cadence cannot see: blocks reaching an OLDER reference through multi-reference prediction --- measured on a block moving 4 pixels per frame, such blocks read as 8 or 16 with source at plus or minus one throughout --- and B-frame vectors pointing at a future reference, which default filtering excludes. So median_dx and median_dy are medians for a reason and are robust to the residuals; magnitude is a sum and inherits what remains of the over-count. Treat magnitude as a quantity to correlate against itself over time, which is what it was validated for, rather than as pixels per second.

Returns:

Name	Type	Description
MgMotionVectorData	MgMotionVectorData	time in seconds, picture_type as 'I'/'P'/'B',
	MgMotionVectorData	n_vectors, magnitude (the area-weighted sum of displacement, which is the
	MgMotionVectorData	quantity to compare against quantity of motion), and median_dx / median_dy
	MgMotionVectorData	(the typical displacement in pixels of the blocks that moved --- taken over
	MgMotionVectorData	every block instead, these would report the still background's zero however fast
	MgMotionVectorData	the one moving thing was going). Frames with no vectors, which includes every
	MgMotionVectorData	intra frame, read as zero throughout.

Raises:

Type	Description
ImportError	if PyAV is not installed. ffprobe reports that the side data exists but will not print the vectors themselves, and ffmpeg has no numeric dump, so there is no route to these numbers through the command line.

Source code in musicalgestures/_motionvectors.py

def mg_motionvectordata(self: "musicalgestures.MgVideo") -> "MgMotionVectorData":
    """Read the motion vectors the codec already computed, as numbers rather than arrows.

    `motionvectors()` renders these as a video to look at. This returns them as data, and
    the difference in cost is the point: an inter-frame codec has already searched for the
    displacement of every macroblock, so reading its answer is close to free, while
    differencing pixels does the same search again in Python. On a 103-minute recording,
    decoding took 2.2 s and decoding with the vectors took 2.7 s, against 27 minutes for
    the frame-differenced quantity of motion.

    **What it is not.** The vectors are the encoder's decisions, not a measurement of the
    scene: an encoder is entitled to any vector that predicts the block cheaply, and on
    flat or still regions it will pick one that has nothing to do with motion. The
    result tracks quantity of motion well where there is motion to track and poorly where
    there is not --- on that same corpus, median r = 0.86 in windows with motion and
    0.57 in windows without. Re-encoded proxies carry their proxy encoder's vectors, not
    the camera's.

    **Vectors are normalised to point forward in time.** A B-frame may predict a block
    from a later frame, which reverses the sign; dividing by the vector's `source`
    corrects that, so a block moving right reads as positive whichever way it was
    predicted. Without it a B-frame's vectors point backwards half the time and averaging
    them gives roughly nothing.

    **The reference distance is corrected from the cadence, with two residuals.**
    `source` records only the direction, plus or minus one, never how many frames away
    the reference was, so the distance is recovered instead from the picture-type
    cadence: past-referencing vectors are divided by the display-order gap to the
    previous reference frame, which makes a P-frame after a run of B-frames read
    per-frame displacement rather than the whole span's. What the cadence cannot see:
    blocks reaching an OLDER reference through multi-reference prediction --- measured
    on a block moving 4 pixels per frame, such blocks read as 8 or 16 with `source` at
    plus or minus one throughout --- and B-frame vectors pointing at a future
    reference, which default filtering excludes. So `median_dx` and `median_dy` are
    medians for a reason and are robust to the residuals; `magnitude` is a sum and
    inherits what remains of the over-count. Treat `magnitude` as a quantity to
    correlate against itself over time, which is what it was validated for, rather
    than as pixels per second.

    Returns:
        MgMotionVectorData: `time` in seconds, `picture_type` as 'I'/'P'/'B',
        `n_vectors`, `magnitude` (the area-weighted sum of displacement, which is the
        quantity to compare against quantity of motion), and `median_dx` / `median_dy`
        (the typical displacement in pixels **of the blocks that moved** --- taken over
        every block instead, these would report the still background's zero however fast
        the one moving thing was going). Frames with no vectors, which includes every
        intra frame, read as zero throughout.

    Raises:
        ImportError: if PyAV is not installed. ffprobe reports that the side data exists
            but will not print the vectors themselves, and ffmpeg has no numeric dump, so
            there is no route to these numbers through the command line.
    """
    try:
        import av
    except ImportError as exc:                       # pragma: no cover - trivial branch
        raise ImportError(
            "Reading motion vectors as data needs PyAV. Install it with "
            "`pip install musicalgestures[motionvectors]`, or `pip install av`. "
            "The rendering version, MgVideo.motionvectors(), needs only ffmpeg."
        ) from exc
    import numpy as np

    container = av.open(self.filename)
    stream = container.streams.video[0]
    stream.thread_type = "AUTO"
    #: AV_CODEC_FLAG2_EXPORT_MVS. PyAV exposes flags2 as a plain int and does not
    #: re-export the constant, so it is written out rather than imported.
    stream.codec_context.flags2 |= 1 << 28

    time: list[float] = []
    kinds: list[str] = []
    counts: list[int] = []
    magnitude: list[float] = []
    med_dx: list[float] = []
    med_dy: list[float] = []
    #: The reference distance, recovered from the cadence. `source` carries only the
    #: SIGN of the reference --- measured on a real encode it is plus or minus one
    #: throughout, whatever the distance --- so a P-frame that follows a run of
    #: B-frames reports its whole multi-frame displacement as one frame's. What IS
    #: recoverable while decoding in display order is the gap back to the previous
    #: reference frame (I or P), and dividing the past-referencing vectors by that gap
    #: corrects the cadence part exactly. Two residuals remain and are documented
    #: rather than hidden: blocks reaching an OLDER reference through multi-reference
    #: prediction, and B-frame vectors pointing at a future reference --- the toolbox
    #: excludes B-frames by default, and their vectors were already the worse signal.
    display_idx = -1
    last_ref_idx = None
    for frame in container.decode(stream):
        display_idx += 1
        kind = _PICTURE_TYPES.get(int(frame.pict_type), "?")
        past_gap = 1.0 if last_ref_idx is None else max(1.0, display_idx - last_ref_idx)
        if kind in ("I", "P"):
            last_ref_idx = display_idx
        time.append(float(frame.pts * stream.time_base) if frame.pts is not None
                    else (time[-1] if time else 0.0))
        kinds.append(kind)
        vectors = frame.side_data.get("MOTION_VECTORS")
        table = vectors.to_ndarray() if vectors is not None else None
        if table is None or len(table) == 0:
            counts.append(0)
            magnitude.append(0.0)
            med_dx.append(0.0)
            med_dy.append(0.0)
            continue
        scale = np.maximum(table["motion_scale"].astype(np.float64), 1)
        #: Two corrections in one division: ffmpeg defines the vector as
        #: `src = dst + motion / motion_scale`, so it points from where the block is
        #: now back to where it came from --- content travelling right carries a
        #: NEGATIVE motion_x --- and `source` is negative for a past reference,
        #: positive for a future one. Dividing by `source` undoes both senses; the
        #: cadence gap above then scales the past-referencing multi-frame predictions
        #: down to one frame.
        source = table["source"].astype(np.float64)
        source[source == 0] = -1
        dx = table["motion_x"].astype(np.float64) / scale / source
        dy = table["motion_y"].astype(np.float64) / scale / source
        past = source < 0
        dx[past] /= past_gap
        dy[past] /= past_gap
        area = table["w"].astype(np.float64) * table["h"]
        counts.append(len(table))
        magnitude.append(float((np.hypot(dx, dy) * area).sum()))
        #: Over the blocks that moved, not over every block. Most of a frame is
        #: usually still, so a median over all blocks is the background's zero however
        #: fast the one moving thing is going, which is not a useful answer to "which
        #: way did it move". Threshold-free: a block counts as moving if the encoder
        #: gave it a displacement at all.
        moving = (dx != 0) | (dy != 0)
        med_dx.append(float(np.median(dx[moving])) if moving.any() else 0.0)
        med_dy.append(float(np.median(dy[moving])) if moving.any() else 0.0)
    codec = stream.codec_context.name
    container.close()

    if counts and not any(counts):
        #: An array of zeros means "nobody moved" and "this codec carries no vectors"
        #: equally well, and the caller cannot tell them apart. ffmpeg exports vectors
        #: for H.264 and MPEG-4 Part 2; HEVC and VP9 decode fine and return none.
        warnings.warn(
            f"No motion vectors in this file: the {codec} stream decoded but carried "
            f"none, so every value below is zero. ffmpeg exports motion vectors for "
            f"H.264 and MPEG-4 Part 2, and not for HEVC, VP9 or any intra-only format. "
            f"Re-encode to H.264 to read them.",
            UserWarning, stacklevel=2)

    return MgMotionVectorData(
        time=np.asarray(time, dtype=np.float64),
        picture_type=np.asarray(kinds),
        n_vectors=np.asarray(counts, dtype=np.int64),
        magnitude=np.asarray(magnitude, dtype=np.float64),
        median_dx=np.asarray(med_dx, dtype=np.float64),
        median_dy=np.asarray(med_dy, dtype=np.float64),
    )

motionvectorhistory

motionvectorhistory(mode='direction', colormap='inferno', gamma=0.5, target_name=None, overwrite=True)

The whole space the motion happened in, accumulated from the codec's vectors.

motionhistory() renders the Bobick--Davis image from frame differences and encodes recency; heatmap() accumulates where pixels changed. Neither can say which way anything went, because a frame difference has no sign --- somebody entering a region and leaving it look identical. A motion vector is a displacement, so this can, and that is the reason for it to exist.

Parameters:

Name	Type	Description	Default
mode	str	'direction' colours each cell by the direction motion took there, hue running around the compass, with brightness for how much motion the cell saw. 'magnitude' drops the direction and applies colormap to the amount alone, which is the comparable view when direction is a distraction. Defaults to 'direction'.	'direction'
colormap	str	Matplotlib colormap, used by 'magnitude'. Defaults to 'inferno'.	'inferno'
gamma	float	Applied to the accumulated amount before colouring, so that a few violent moments do not leave everything else black. Defaults to 0.5.	0.5
target_name	str	Output path. Defaults to the input name with _motionvectorhistory.	None
overwrite	bool	Defaults to True.	True

Returns:

Name	Type	Description
MgImage	MgImage	the rendered image, at the video's own dimensions.

Notes

Built from P-frames only, for the reason given on motionvectordata. And the vectors are the encoder's decisions: over a region with no detail every candidate predicts equally well, so the encoder copies whatever its neighbours used and the field there is inherited rather than observed. Expect that wherever the picture is flat.

Source code in musicalgestures/_motionvectors.py

def mg_motionvectorhistory(self: "musicalgestures.MgVideo", mode: str = "direction",
                           colormap: str = "inferno", gamma: float = 0.5,
                           target_name: str | None = None,
                           overwrite: bool = True) -> "MgImage":
    """The whole space the motion happened in, accumulated from the codec's vectors.

    `motionhistory()` renders the Bobick--Davis image from frame differences and encodes
    *recency*; `heatmap()` accumulates where pixels changed. Neither can say which way
    anything went, because a frame difference has no sign --- somebody entering a region
    and leaving it look identical. A motion vector is a displacement, so this can, and
    that is the reason for it to exist.

    Args:
        mode (str, optional): `'direction'` colours each cell by the direction motion
            took there, hue running around the compass, with brightness for how much
            motion the cell saw. `'magnitude'` drops the direction and applies `colormap`
            to the amount alone, which is the comparable view when direction is a
            distraction. Defaults to `'direction'`.
        colormap (str, optional): Matplotlib colormap, used by `'magnitude'`. Defaults to
            `'inferno'`.
        gamma (float, optional): Applied to the accumulated amount before colouring, so
            that a few violent moments do not leave everything else black. Defaults to
            0.5.
        target_name (str, optional): Output path. Defaults to the input name with
            `_motionvectorhistory`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgImage: the rendered image, at the video's own dimensions.

    Notes:
        Built from P-frames only, for the reason given on `motionvectordata`. And the
        vectors are the encoder's decisions: over a region with no detail every candidate
        predicts equally well, so the encoder copies whatever its neighbours used and the
        field there is inherited rather than observed. Expect that wherever the picture is
        flat.
    """
    import cv2
    import matplotlib
    import numpy as np

    from musicalgestures._utils import MgImage

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_motionvectorhistory.png', target_name, overwrite)

    weight, vx, vy, coherence = accumulate_motion_vectors(self.filename)
    #: Normalised against a high percentile, not the maximum. Over a long recording the
    #: distribution is heavy-tailed and dividing by the maximum leaves the typical cell
    #: close to it, so the whole frame comes out uniformly bright and shows nothing.
    ceiling = np.percentile(weight[weight > 0], 99) if (weight > 0).any() else 1.0
    amount = np.clip(weight / max(ceiling, 1e-9), 0, 1)
    if gamma and gamma != 1.0:
        amount = np.power(amount, gamma)

    if mode == "direction":
        #: Hue is the compass bearing, so opposite directions are opposite colours.
        #: Value is how much motion there was, and **saturation is how consistently it
        #: went that way**. Pinning saturation at full was the first version's mistake:
        #: it painted every cell a vivid colour whether or not the direction meant
        #: anything, and since every cell accumulates something over a hundred minutes,
        #: the result was a uniform wash. Grey now reads as "moved, but not any one
        #: way", which is the honest thing to say about most of a room.
        hue = (np.arctan2(vy, vx) + np.pi) / (2 * np.pi) * 179
        hsv = np.stack([hue.astype(np.uint8),
                        np.clip(coherence * 255, 0, 255).astype(np.uint8),
                        np.clip(amount * 255, 0, 255).astype(np.uint8)], axis=-1)
        rgb = cv2.cvtColor(hsv, cv2.COLOR_HSV2RGB)
    elif mode == "magnitude":
        cmap = matplotlib.colormaps[colormap]
        rgb = (cmap(amount)[..., :3] * 255).astype(np.uint8)
    elif mode == "coherence":
        cmap = matplotlib.colormaps[colormap]
        rgb = (cmap(coherence)[..., :3] * 255).astype(np.uint8)
    else:
        raise ValueError(
            f"mode must be 'direction', 'magnitude' or 'coherence', not {mode!r}")

    width, height = self.width, self.height
    full = cv2.resize(rgb, (width, height), interpolation=cv2.INTER_NEAREST)
    cv2.imwrite(target_name, full[..., ::-1])

    self.motionvectorhistory_image = MgImage(target_name)
    return self.motionvectorhistory_image

motionvectorgrams

motionvectorgrams(colormap='inferno', gamma=0.5, max_width=4000, target_name=None, overwrite=True)

Motiongrams built from the codec's motion vectors.

The same view motiongrams() gives, at a fraction of the cost, because the displacement has already been computed by the encoder. The trade is resolution: this is drawn on the macroblock lattice, sixteen pixels, and on P-frames only, so on 50 fps footage it carries about 12.6 columns per second rather than 50.

Parameters:

Name	Type	Description	Default
colormap	str	Matplotlib colormap. Defaults to 'inferno'.	'inferno'
gamma	float	Applied before colouring so quiet passages stay visible. Defaults to 0.5.	0.5
max_width	int	Longest the saved image's TIME axis may be, in pixels --- across the horizontal image, down the vertical one. One point per P-frame is right for the array and wrong for the picture: a 158-minute session has 77,592 of them, and stretching that to the video's height gave a 149-megapixel PNG of 309 MB for something meant to be glanced at. Points are pooled by taking the maximum, not the mean, so a brief accent still shows in a column standing for several seconds. Defaults to 4000.	4000
target_name	str	Output path; the two images take _mvgram_y and _mvgram_x from it, named by the position axis each keeps. Defaults to the input name.	None
overwrite	bool	Defaults to True.	True

Returns:

Name	Type	Description
MgList	MgList	the y-motiongram and the x-motiongram, in that order, in the classic
	MgList	orientations --- y wide with time rightward, x tall with time downward.

Source code in musicalgestures/_motionvectors.py

def mg_motionvectorgrams(self: "musicalgestures.MgVideo", colormap: str = "inferno",
                         gamma: float = 0.5, max_width: int = 4000,
                         target_name: str | None = None,
                         overwrite: bool = True) -> "MgList":
    """Motiongrams built from the codec's motion vectors.

    The same view `motiongrams()` gives, at a fraction of the cost, because the
    displacement has already been computed by the encoder. The trade is resolution: this
    is drawn on the macroblock lattice, sixteen pixels, and on P-frames only, so on 50 fps
    footage it carries about 12.6 columns per second rather than 50.

    Args:
        colormap (str, optional): Matplotlib colormap. Defaults to `'inferno'`.
        gamma (float, optional): Applied before colouring so quiet passages stay visible.
            Defaults to 0.5.
        max_width (int, optional): Longest the saved image's TIME axis may be, in
            pixels --- across the horizontal image, down the vertical one. One point per
            P-frame is right for the array and wrong for the picture: a 158-minute
            session has 77,592 of them, and stretching that to the video's height gave a
            149-megapixel PNG of 309 MB for something meant to be glanced at. Points are
            pooled by taking the maximum, not the mean, so a brief accent still shows in
            a column standing for several seconds. Defaults to 4000.
        target_name (str, optional): Output path; the two images take `_mvgram_y` and
            `_mvgram_x` from it, named by the position axis each keeps. Defaults to
            the input name.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgList: the y-motiongram and the x-motiongram, in that order, in the classic
        orientations --- y wide with time rightward, x tall with time downward.
    """
    import cv2
    import matplotlib
    import numpy as np

    from musicalgestures._mglist import MgList
    from musicalgestures._utils import MgImage

    of, fex = os.path.splitext(self.filename)
    base = resolve_filename(of, '_mvgram.png', target_name, overwrite)
    stem, _ = os.path.splitext(base)

    horizontal, vertical = motion_vector_motiongrams(self.filename)
    cmap = matplotlib.colormaps[colormap]
    images = []
    #: Classic orientations, named for the picture: the horizontal gram is wide with
    #: time rightward, the vertical gram is TALL with time downward. The time axis is
    #: axis 1 of the first and axis 0 of the second, so everything below works on
    #: (space, time) and transposes the vertical picture back at the end.
    for gram, suffix, span, tall in ((horizontal, "_y", self.height, False),
                                     (vertical.T, "_x", self.width, True)):
        if gram.size and gram.shape[1] > max_width:
            #: Pool by maximum rather than mean. A column here stands for several seconds
            #: once a long recording is squeezed to a readable length, and averaging
            #: would bury a one-second accent under the quiet around it --- which is
            #: exactly what a motiongram is read for.
            edges = np.linspace(0, gram.shape[1], max_width + 1).astype(int)
            gram = np.stack([gram[:, a:max(b, a + 1)].max(axis=1)
                             for a, b in zip(edges[:-1], edges[1:])], axis=1)
        scaled = gram / gram.max() if gram.size and gram.max() > 0 else gram
        if gamma and gamma != 1.0:
            scaled = np.power(scaled, gamma)
        rgb = (cmap(scaled)[..., :3] * 255).astype(np.uint8)
        #: Stretched back to the frame's own dimension on the spatial axis so the two
        #: grams line up with the video and with each other; nearest, because smoothing a
        #: macroblock lattice invents detail the vectors never had.
        out = cv2.resize(rgb, (rgb.shape[1], span), interpolation=cv2.INTER_NEAREST)
        if tall:
            out = out.transpose(1, 0, 2)
        path = f"{stem}{suffix}.png"
        cv2.imwrite(path, out[..., ::-1])
        images.append(MgImage(path))

    self.motionvectorgrams_images = MgList(images)
    return self.motionvectorgrams_images

motionvectorwaterfall

motionvectorwaterfall(n_samples=40, axis='horizontal', cmap='viridis', dpi=200, elev=35, azim=-60, target_name=None, overwrite=True)

A waterfall of motion profiles, cascading through time.

The same cascade silhouette_waterfall() draws, but from displacement rather than from a silhouette, so it needs no background subtraction and no pose model --- and it shows where motion was, not where a body was. A dancer standing still has a silhouette and no vectors.

Parameters:

Name	Type	Description	Default
n_samples	int	How many profiles to stack. Defaults to 40.	40
axis	str	'horizontal' profiles over x, 'vertical' over y. Defaults to 'horizontal'.	'horizontal'
cmap	str	Matplotlib colormap, applied over time. Defaults to 'viridis'.	'viridis'
dpi	int	Defaults to 200.	200
elev, azim	float	3D view angles. Default to 35 and -60.	required
target_name	str	Output path. Defaults to the input name with _motionvectorwaterfall.	None
overwrite	bool	Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	MgFigure	the rendered waterfall.

Source code in musicalgestures/_motionvectors.py

def mg_motionvectorwaterfall(self: "musicalgestures.MgVideo", n_samples: int = 40,
                             axis: str = "horizontal", cmap: str = "viridis",
                             dpi: int = 200, elev: float = 35, azim: float = -60,
                             target_name: str | None = None,
                             overwrite: bool = True) -> "MgFigure":
    """A waterfall of motion profiles, cascading through time.

    The same cascade `silhouette_waterfall()` draws, but from displacement rather than
    from a silhouette, so it needs no background subtraction and no pose model --- and it
    shows where motion was, not where a body was. A dancer standing still has a
    silhouette and no vectors.

    Args:
        n_samples (int, optional): How many profiles to stack. Defaults to 40.
        axis (str, optional): `'horizontal'` profiles over x, `'vertical'` over y.
            Defaults to `'horizontal'`.
        cmap (str, optional): Matplotlib colormap, applied over time. Defaults to
            `'viridis'`.
        dpi (int, optional): Defaults to 200.
        elev, azim (float, optional): 3D view angles. Default to 35 and -60.
        target_name (str, optional): Output path. Defaults to the input name with
            `_motionvectorwaterfall`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgFigure: the rendered waterfall.
    """
    import matplotlib
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._utils import MgFigure

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_motionvectorwaterfall.png', target_name,
                                   overwrite)

    profiles, times = motion_vector_profiles(self.filename, n_samples=n_samples,
                                             axis=axis)
    fig = plt.figure(figsize=(10, 6), dpi=dpi)
    ax = fig.add_subplot(111, projection="3d")
    colours = matplotlib.colormaps[cmap](np.linspace(0, 1, max(len(profiles), 1)))
    x = np.arange(profiles.shape[1]) if profiles.size else np.arange(0)
    for i, profile in enumerate(profiles):
        ax.plot(x, np.full_like(x, times[i], dtype=float), profile,
                color=colours[i], linewidth=0.9)
    ax.set_xlabel("x" if axis == "horizontal" else "y")
    ax.set_ylabel("time (s)")
    ax.set_zlabel("motion")
    ax.view_init(elev=elev, azim=azim)
    fig.tight_layout()
    fig.savefig(target_name)
    plt.close(fig)

    self.motionvectorwaterfall_figure = MgFigure(
        figure=None, figure_type="video.motionvectorwaterfall",
        data={"profiles": profiles, "times": times}, layers=None,
        image=target_name)
    return self.motionvectorwaterfall_figure

motionvectoroverview

motionvectoroverview(colormap='inferno', gamma=0.5, target_name=None, overwrite=True)

One sheet per recording, from one decode: where, how much, and when.

Built for a first look at unfamiliar material rather than for measurement. Hours of archive video, several representations per file, as the starting point for qualitative annotation --- so it favours covering the whole recording cheaply over resolving any part of it finely.

Five panels, of the three kinds a first pass wants:

• spatial --- the area motion covered, coloured by direction, and the same accumulation as plain amount;
• temporal --- motion over the whole recording, the curve to scrub against;
• spatio-temporal --- the y- and x-motiongrams, position against time, where a body crossing the room draws a diagonal.

Everything comes from the codec's own motion vectors, so the cost is close to the cost of reading the file: about five minutes for a 100-minute 1920x1080 recording, against roughly twenty-four for the same views computed one at a time, and far less than differencing the pixels.

Parameters:

Name	Type	Description	Default
colormap	str	Matplotlib colormap for the amount panels. Defaults to 'inferno'.	'inferno'
gamma	float	Applied before colouring so quiet passages stay visible. Defaults to 0.5.	0.5
target_name	str	Output path. Defaults to the input name with _motionvectoroverview.	None
overwrite	bool	Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	MgFigure	the sheet, with every underlying array in .data so the same pass can
	MgFigure	feed further analysis without decoding again.

Source code in musicalgestures/_motionvectors.py

def mg_motionvectoroverview(self: "musicalgestures.MgVideo", colormap: str = "inferno",
                            gamma: float = 0.5, target_name: str | None = None,
                            overwrite: bool = True) -> "MgFigure":
    """One sheet per recording, from one decode: where, how much, and when.

    Built for a first look at unfamiliar material rather than for measurement. Hours of
    archive video, several representations per file, as the starting point for
    qualitative annotation --- so it favours covering the whole recording cheaply over
    resolving any part of it finely.

    Five panels, of the three kinds a first pass wants:

    * **spatial** --- the area motion covered, coloured by direction, and the same
      accumulation as plain amount;
    * **temporal** --- motion over the whole recording, the curve to scrub against;
    * **spatio-temporal** --- the y- and x-motiongrams, position against time,
      where a body crossing the room draws a diagonal.

    Everything comes from the codec's own motion vectors, so the cost is close to the
    cost of reading the file: about five minutes for a 100-minute 1920x1080 recording,
    against roughly twenty-four for the same views computed one at a time, and far less
    than differencing the pixels.

    Args:
        colormap (str, optional): Matplotlib colormap for the amount panels. Defaults to
            `'inferno'`.
        gamma (float, optional): Applied before colouring so quiet passages stay visible.
            Defaults to 0.5.
        target_name (str, optional): Output path. Defaults to the input name with
            `_motionvectoroverview`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgFigure: the sheet, with every underlying array in `.data` so the same pass can
        feed further analysis without decoding again.
    """
    import cv2
    import matplotlib
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._utils import MgFigure

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_motionvectoroverview.png', target_name,
                                   overwrite)
    v = motion_vector_views(self.filename)

    def shade(a):
        ceiling = np.percentile(a[a > 0], 99) if (a > 0).any() else 1.0
        return np.power(np.clip(a / max(ceiling, 1e-9), 0, 1), gamma)

    hue = (np.arctan2(v.history_vy, v.history_vx) + np.pi) / (2 * np.pi) * 179
    direction = cv2.cvtColor(np.stack([
        hue.astype(np.uint8),
        np.clip(v.history_coherence * 255, 0, 255).astype(np.uint8),
        np.clip(shade(v.history_weight) * 255, 0, 255).astype(np.uint8)], axis=-1),
        cv2.COLOR_HSV2RGB)

    cmap = matplotlib.colormaps[colormap]
    fig = plt.figure(figsize=(16, 11), dpi=110)
    grid = fig.add_gridspec(3, 2, height_ratios=[3, 1, 2], hspace=0.28, wspace=0.12)

    ax = fig.add_subplot(grid[0, 0])
    ax.imshow(direction, aspect="auto", interpolation="nearest")
    ax.set_title("where, and which way (hue = direction, saturation = consistency)")
    ax.set_xticks([]); ax.set_yticks([])

    ax = fig.add_subplot(grid[0, 1])
    ax.imshow(shade(v.history_weight), cmap=colormap, aspect="auto",
              interpolation="nearest")
    ax.set_title("where, as amount")
    ax.set_xticks([]); ax.set_yticks([])

    ax = fig.add_subplot(grid[1, :])
    if len(v.time):
        ax.plot(v.time / 60, v.magnitude, linewidth=0.4, color="#c1440e")
        ax.set_xlim(v.time[0] / 60, v.time[-1] / 60)
    ax.set_title("how much, over the recording")
    ax.set_xlabel("minutes")
    ax.set_yticks([])

    minutes = (v.time[-1] / 60) if len(v.time) else 1
    #: Classic orientations, named for the picture: the horizontal gram wide with time
    #: rightward, the vertical gram with time running DOWNWARD in its panel.
    ax = fig.add_subplot(grid[2, 0])
    if v.motiongram_y.size:
        ax.imshow(shade(v.motiongram_y), cmap=colormap, aspect="auto",
                  interpolation="nearest",
                  extent=(0, minutes, v.motiongram_y.shape[0], 0))
    ax.set_title("y-motiongram (y down, time \u2192)")
    ax.set_xlabel("minutes")
    ax.set_yticks([])

    ax = fig.add_subplot(grid[2, 1])
    if v.motiongram_x.size:
        ax.imshow(shade(v.motiongram_x), cmap=colormap, aspect="auto",
                  interpolation="nearest",
                  extent=(0, v.motiongram_x.shape[1], minutes, 0))
    ax.set_title("x-motiongram (x across, time \u2193)")
    ax.set_ylabel("minutes")
    ax.set_xticks([])

    fig.suptitle(os.path.basename(self.filename), fontsize=11)
    fig.savefig(target_name, bbox_inches="tight")
    plt.close(fig)

    self.motionvectoroverview_figure = MgFigure(
        figure=None, figure_type="video.motionvectoroverview",
        data={"time": v.time, "magnitude": v.magnitude,
              "history_weight": v.history_weight, "history_vx": v.history_vx,
              "history_vy": v.history_vy, "history_coherence": v.history_coherence,
              "motiongram_y": v.motiongram_y,
              "motiongram_x": v.motiongram_x},
        layers=None, image=target_name)
    return self.motionvectoroverview_figure

motionscape

motionscape(track=None, track_times=None, n_scales=96, colormap='magma', gamma=0.5, target_name=None, overwrite=True)

A scape of the recording's motion: every timescale in one triangle.

After Craig Sapp's keyscape, where each row is a window length and the apex is a single window covering the whole piece. Here each cell is the mean quantity of motion over that window, so the plot answers a question the flat curve cannot: at what scale does this recording stop looking like one thing? A session of even, continuous improvisation is flat all the way up. A session of separated bursts stays separated at the base and merges into one mass near the apex, and the height at which it merges is the length of the structure.

Parameters:

Name	Type	Description	Default
track	array - like	One value per frame to build the scape from. Pass a frame-differenced quantity of motion here --- mg_motion's QomRaw, or a qom.f4 read off disk --- to scape that instead. Nothing about the construction assumes where the numbers came from. Defaults to None, which reads the codec's motion vectors, cheap enough to run over an archive.	None
track_times	array - like	Seconds for each value in track. Needed whenever the track is not one value per video frame --- notably motionvectoroverview's own magnitude, which is P-frames only and so arrives at about a quarter of the frame rate. Without it the time axis is wrong by that factor while looking perfectly reasonable. Defaults to None, meaning the track is at the video's frame rate.	None
n_scales	int	Rows in the triangle. Defaults to 96.	96
colormap	str	Defaults to 'magma'.	'magma'
gamma	float	Applied before colouring. Defaults to 0.5.	0.5
target_name	str	Output path. Defaults to the input name with _motionscape.	None
overwrite	bool	Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	MgFigure	the scape, with the track and the triangle in .data.

Source code in musicalgestures/_motionvectors.py

def mg_motionscape(self: "musicalgestures.MgVideo", track=None, track_times=None,
                   n_scales: int = 96, colormap: str = "magma",
                   gamma: float = 0.5, target_name: str | None = None,
                   overwrite: bool = True) -> "MgFigure":
    """A scape of the recording's motion: every timescale in one triangle.

    After Craig Sapp's keyscape, where each row is a window length and the apex is a
    single window covering the whole piece. Here each cell is the mean quantity of motion
    over that window, so the plot answers a question the flat curve cannot: **at what
    scale does this recording stop looking like one thing?** A session of even, continuous
    improvisation is flat all the way up. A session of separated bursts stays separated at
    the base and merges into one mass near the apex, and the height at which it merges is
    the length of the structure.

    Args:
        track (array-like, optional): One value per frame to build the scape from. Pass
            a frame-differenced quantity of motion here --- `mg_motion`'s `QomRaw`, or a
            `qom.f4` read off disk --- to scape that instead. Nothing about the
            construction assumes where the numbers came from. Defaults to None, which
            reads the codec's motion vectors, cheap enough to run over an archive.
        track_times (array-like, optional): Seconds for each value in `track`. Needed
            whenever the track is not one value per video frame --- notably
            `motionvectoroverview`'s own `magnitude`, which is P-frames only and so
            arrives at about a quarter of the frame rate. Without it the time axis is
            wrong by that factor while looking perfectly reasonable. Defaults to None,
            meaning the track is at the video's frame rate.
        n_scales (int, optional): Rows in the triangle. Defaults to 96.
        colormap (str, optional): Defaults to `'magma'`.
        gamma (float, optional): Applied before colouring. Defaults to 0.5.
        target_name (str, optional): Output path. Defaults to the input name with
            `_motionscape`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgFigure: the scape, with the track and the triangle in `.data`.
    """
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._utils import MgFigure

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_motionscape.png', target_name, overwrite)

    if track is not None:
        track = np.asarray(track, dtype=np.float64).ravel()
        if track_times is not None:
            times = np.asarray(track_times, dtype=np.float64).ravel()
            if len(times) != len(track):
                raise ValueError(
                    f"track_times has {len(times)} values and track has {len(track)}; "
                    "they must describe the same series")
        else:
            times = np.arange(len(track)) / float(self.fps)
        used = "supplied track"
    else:
        views = motion_vector_views(self.filename)
        track, times, used = views.magnitude, views.time, "motion vectors"

    scape = motion_scape(track, n_scales=n_scales)
    shown = scape / np.nanpercentile(scape, 99) if np.isfinite(scape).any() else scape
    shown = np.power(np.clip(shown, 0, 1), gamma)

    minutes = (times[-1] / 60) if len(times) else 1
    fig, ax = plt.subplots(figsize=(14, 7), dpi=120)
    ax.imshow(shown, cmap=colormap, aspect="auto", interpolation="nearest",
              extent=(0, minutes, 0, n_scales))
    ax.set_xlabel("minutes")
    ax.set_ylabel("window length: short at the base, whole recording at the apex")
    ax.set_yticks([])
    ax.set_title(f"{os.path.basename(self.filename)} --- motion scape ({used})")
    fig.tight_layout()
    fig.savefig(target_name)
    plt.close(fig)

    self.motionscape_figure = MgFigure(
        figure=None, figure_type="video.motionscape",
        data={"track": track, "time": times, "scape": scape, "source": used},
        layers=None, image=target_name)
    return self.motionscape_figure

pose_timeline

pose_timeline(view='strip', n_samples=12, min_visibility=0.5, landmarks=None, times=None, cmap='viridis', dpi=200, trajectories=None, markers=TRAJECTORY_GROUPS, smooth=SMOOTH, smooth_spatial=SMOOTH_SPATIAL, target_name=None, overwrite=True, **pose_kwargs)

Postures and trajectories over time. See pose_timeline.

Landmarks come from a cached pose() result when there is one, exactly as posegram resolves them, and otherwise from a fresh extraction.

Source code in musicalgestures/_posetimeline.py

def mg_pose_timeline(self: "musicalgestures.MgVideo", view: str = "strip",
                     n_samples: int = 12, min_visibility: float = 0.5,
                     landmarks=None, times=None, cmap: str = "viridis", dpi: int = 200,
                     trajectories=None, markers=TRAJECTORY_GROUPS, smooth: int = SMOOTH,
                     smooth_spatial: int = SMOOTH_SPATIAL,
                     target_name: str | None = None, overwrite: bool = True,
                     **pose_kwargs) -> "MgFigure":
    """Postures and trajectories over time. See `pose_timeline`.

    Landmarks come from a cached `pose()` result when there is one, exactly as `posegram`
    resolves them, and otherwise from a fresh extraction.
    """
    import matplotlib.pyplot as plt

    from musicalgestures._utils import MgFigure, resolve_filename

    if landmarks is None:
        from musicalgestures._posetools import extract_pose_landmarks
        result = extract_pose_landmarks(self.filename, quiet=True, verbose=False,
                                        **pose_kwargs)
        landmarks = result["landmarks"]
        #: The key is `time`, not `times`. Asking for the wrong one silently gave frame
        #: numbers on an axis labelled seconds, which is a plausible-looking wrong answer.
        times = result.get("time")
        if times is None and result.get("fps"):
            times = np.arange(len(landmarks)) / float(result["fps"])

    target_name = resolve_filename(self.of, f"_posetimeline_{view}.png", target_name,
                                   overwrite)
    figure = pose_timeline(landmarks, view=view, n_samples=n_samples,
                           min_visibility=min_visibility, times=times,
                           width=self.width, height=self.height, cmap=cmap, dpi=dpi,
                           trajectories=trajectories, markers=markers, smooth=smooth,
                           smooth_spatial=smooth_spatial)
    figure.savefig(target_name, dpi=dpi)
    plt.close(figure)
    self.pose_timeline_figure = MgFigure(
        figure=figure, figure_type="video.posetimeline",
        data={"view": view}, layers=None, image=target_name)
    return self.pose_timeline_figure

multishot

multishot(n_bodies=8, n_candidates=120, start=None, end=None, width=960, tolerance=TOLERANCE, feather=FEATHER, min_area=MIN_AREA, max_area=MAX_AREA, max_border=MAX_BORDER, shadow_ratio=SHADOW_RATIO, region=None, segmenter='plate', select='spaced', background='plate', colorize=False, animate=False, frame_ms=700, target_name=None, overwrite=True)

Many moments of this recording in one picture, as an MgImage.

The method form of multishot, so it composes with the rest of the object API. See that function for what the arguments mean and for how this differs from stroboscope().

Parameters:

Name	Type	Description	Default
animate	bool	Write a looping GIF of the build-up instead of the still: the bare room first, one more body per frame in time order, and the finished composite held for three frames' worth at the end.	False
frame_ms	int	Milliseconds per animation frame. Defaults to 700.	700
target_name	str	Output name. Defaults to "_multishot.png", or "_multishot.gif" when animate=True.	None
overwrite	bool	Overwrite or auto-increment. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	the composite, or the GIF of it building up.

Raises:

Type	Description
ValueError	when no frame held a body of a plausible size. An empty room is not a result, and returning one silently would look like the recording was empty rather than like the size bounds matched nothing.

Source code in musicalgestures/_multishot.py

def mg_multishot(self: "musicalgestures.MgVideo", n_bodies: int = 8,
                 n_candidates: int = 120, start=None, end=None, width: int = 960,
                 tolerance: float = TOLERANCE, feather: int = FEATHER,
                 min_area: float = MIN_AREA, max_area: float = MAX_AREA,
                 max_border: float = MAX_BORDER, shadow_ratio: float = SHADOW_RATIO,
                 region=None, segmenter: str = "plate", select: str = "spaced",
                 background: str = "plate", colorize: bool = False,
                 animate: bool = False, frame_ms: int = 700,
                 target_name: str | None = None, overwrite: bool = True) -> "MgImage":
    """Many moments of this recording in one picture, as an `MgImage`.

    The method form of `multishot`, so it composes with the rest of the object API. See
    that function for what the arguments mean and for how this differs from
    `stroboscope()`.

    Args:
        animate (bool): Write a looping GIF of the build-up instead of the still: the
            bare room first, one more body per frame in time order, and the finished
            composite held for three frames' worth at the end.
        frame_ms (int): Milliseconds per animation frame. Defaults to 700.
        target_name (str, optional): Output name. Defaults to "_multishot.png", or
            "_multishot.gif" when `animate=True`.
        overwrite (bool, optional): Overwrite or auto-increment. Defaults to True.

    Returns:
        MgImage: the composite, or the GIF of it building up.

    Raises:
        ValueError: when no frame held a body of a plausible size. An empty room is not
            a result, and returning one silently would look like the recording was empty
            rather than like the size bounds matched nothing.
    """
    import cv2

    from musicalgestures._utils import MgImage, resolve_filename

    suffix = "_multishot.gif" if animate else "_multishot.png"
    target_name = resolve_filename(self.of, suffix, target_name, overwrite)
    picture, _ = multishot(self.filename, n_bodies=n_bodies, n_candidates=n_candidates,
                           start=start, end=end, width=width, tolerance=tolerance,
                           feather=feather, min_area=min_area, max_area=max_area,
                           max_border=max_border, shadow_ratio=shadow_ratio,
                           region=region, segmenter=segmenter, select=select,
                           background=background, colorize=colorize, animate=animate)
    if picture is None:
        raise ValueError(
            f"no frame of {self.filename} held a body covering between "
            f"{min_area * 100:.1f} and {max_area * 100:.1f} per cent of the frame. "
            f"Move `min_area`/`max_area` if the framing differs from a whole person at "
            f"studio distance.")
    if animate:
        from PIL import Image

        #: BGR arrays -> RGB stills; the last frame holds so the finished composite
        #: can be read before the loop restarts.
        stills = [Image.fromarray(f[..., ::-1]) for f in picture]
        stills[0].save(target_name, save_all=True, append_images=stills[1:],
                       duration=[frame_ms] * (len(stills) - 1) + [frame_ms * 3],
                       loop=0)
    else:
        cv2.imwrite(target_name, picture)
    self.multishot_image = MgImage(target_name)
    return self.multishot_image

plate

plate(width=960, n_samples=300, stratify=True, target_name=None, overwrite=True)

The empty room this recording was made in, as an MgImage.

The method form of room_plate: a per-pixel median over sampled frames, twice, with the second pass spread over the recording so that anything standing there through a break does not become furniture.

In colour, from the frames the grey pass chose --- not a fresh draw, which would be a different room.

Parameters:

Name	Type	Description	Default
width	int	Working width. Defaults to 960; pass the video's own width for a full-resolution room.	960
n_samples	int	Frames to sample. Defaults to 300.	300
stratify	bool	Spread the second pass over the recording. Defaults to True.	True
target_name	str	Output name. Defaults to "_plate.png".	None
overwrite	bool	Overwrite or auto-increment. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	the room.

Source code in musicalgestures/_multishot.py

def mg_plate(self: "musicalgestures.MgVideo", width: int = 960, n_samples: int = 300,
             stratify: bool = True, target_name: str | None = None,
             overwrite: bool = True) -> "MgImage":
    """The empty room this recording was made in, as an `MgImage`.

    The method form of `room_plate`: a per-pixel median over sampled frames, twice, with
    the second pass spread over the recording so that anything standing there through a
    break does not become furniture.

    **In colour, from the frames the grey pass chose** --- not a fresh draw, which would
    be a different room.

    Args:
        width (int): Working width. Defaults to 960; pass the video's own width for a
            full-resolution room.
        n_samples (int): Frames to sample. Defaults to 300.
        stratify (bool): Spread the second pass over the recording. Defaults to True.
        target_name (str, optional): Output name. Defaults to "_plate.png".
        overwrite (bool, optional): Overwrite or auto-increment. Defaults to True.

    Returns:
        MgImage: the room.
    """
    import cv2

    from musicalgestures._plate import room_plate
    from musicalgestures._utils import MgImage, resolve_filename

    target_name = resolve_filename(self.of, "_plate.png", target_name, overwrite)
    _, used = room_plate(self.filename, n_samples=n_samples, width=width,
                         stratify=stratify)
    capture = cv2.VideoCapture(self.filename)
    stack = []
    for i in used:
        capture.set(cv2.CAP_PROP_POS_FRAMES, int(i))
        ok, frame = capture.read()
        if ok:
            h = max(1, int(frame.shape[0] * width / frame.shape[1]))
            stack.append(cv2.resize(frame, (width, h)))
    capture.release()
    if not stack:
        raise ValueError(f"no frames could be read from {self.filename}")
    plate = np.median(np.asarray(stack, dtype=np.float32), axis=0).astype(np.uint8)
    cv2.imwrite(target_name, plate)
    self.plate_image = MgImage(target_name)
    return self.plate_image

zoompage

zoompage(target_name=None, overwrite=True, max_points=8000, which='videogram_v')

The zoomable page for this recording, in one call, as a method.

Everything derives from the video itself. The motion track and gram come from extract_tracks, computed on first call and cached beside the video like every other analysis; the audio band comes from the video's own soundtrack when it has one; and the player is the video, referenced by its bare name, so the page works from the folder the two share and needs no server.

Parameters:

Name	Type	Description	Default
target_name	str	Output path. Defaults to "_zoom.html" beside the video.	None
overwrite	bool	Overwrite or auto-increment. Defaults to True.	True
max_points	int	Ceiling on embedded envelope points.	8000
which	str	Which cached gram to embed as the strip.	'videogram_v'

Returns:

Name	Type	Description
Path		The file written.

Source code in musicalgestures/_zoomview.py

def mg_zoompage(self, target_name=None, overwrite: bool = True,
                max_points: int = 8000, which: str = "videogram_v"):
    """The zoomable page for this recording, in one call, as a method.

    Everything derives from the video itself. The motion track and gram come from
    `extract_tracks`, computed on first call and cached beside the video like every
    other analysis; the audio band comes from the video's own soundtrack when it has
    one; and the player is the video, referenced by its bare name, so the page works
    from the folder the two share and needs no server.

    Args:
        target_name (str, optional): Output path. Defaults to "_zoom.html" beside
            the video.
        overwrite (bool, optional): Overwrite or auto-increment. Defaults to True.
        max_points (int): Ceiling on embedded envelope points.
        which (str): Which cached gram to embed as the strip.

    Returns:
        Path: The file written.
    """
    import os
    import subprocess

    from musicalgestures._tracks import extract_tracks
    from musicalgestures._utils import resolve_filename

    of, _ = os.path.splitext(self.filename)
    target = resolve_filename(of, "_zoom.html", target_name, overwrite)

    analysis = Path(self.filename).parent / "analysis" / Path(of).name
    if not (analysis / "tracks.json").exists():
        extract_tracks(self.filename, progress=False)
    meta = json.loads((analysis / "tracks.json").read_text())
    duration = int(meta["frames"]) / float(meta["fps"])

    #: The audio band only when the file carries sound; a silent clip gets a page
    #: without one rather than an error.
    probe = subprocess.run(
        ["ffprobe", "-v", "error", "-select_streams", "a", "-show_entries",
         "stream=codec_type", "-of", "csv=p=0", self.filename],
        capture_output=True, text=True)
    has_audio = "audio" in probe.stdout

    return zoomable_page(
        analysis, duration, target, max_points=max_points, which=which,
        audio=self.filename if has_audio else None,
        player=os.path.basename(self.filename),
        title=Path(of).name)

eulerian

eulerian(mode='color', freq_low=0.83, freq_high=1.0, amplification=50, levels=4, chroma_attenuation=1.0, lambda_cutoff=16, target_name=None, overwrite=True)

Applies Eulerian Video Magnification (EVM) to reveal subtle changes in a video.

EVM amplifies small temporal variations that are normally invisible. Two modes are available:

• mode='color' — amplifies subtle colour changes (e.g. blood flow / pulse, breathing). Uses a Gaussian pyramid and an ideal (FFT) temporal band-pass filter. Processed in two passes so only a small down-sampled stack is held in memory.
• mode='motion' — amplifies subtle motion. Uses a Laplacian pyramid with a streaming IIR temporal band-pass filter and spatial-wavelength attenuation, so it runs frame-by-frame with low memory use.

Based on Wu et al., "Eulerian Video Magnification for Revealing Subtle Changes in the World" (SIGGRAPH 2012).

Parameters:

Name	Type	Description	Default
mode	str	'color' or 'motion'. Defaults to 'color'.	'color'
freq_low	float	Lower temporal cutoff in Hz. Defaults to 0.83 (~50 bpm).	0.83
freq_high	float	Upper temporal cutoff in Hz. Defaults to 1.0 (~60 bpm).	1.0
amplification	float	Amplification factor (alpha). Defaults to 50.	50
levels	int	Number of spatial pyramid levels. Defaults to 4.	4
chroma_attenuation	float	Chrominance attenuation in [0, 1] (color mode). Lower values reduce colour artefacts. Defaults to 1.0.	1.0
lambda_cutoff	float	Spatial wavelength cutoff for amplitude attenuation (motion mode). Defaults to 16.	16
target_name	str	Target output name. Defaults to None (input filename with the suffix "_evm").	None
overwrite	bool	Whether to allow overwriting or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	MgVideo	An MgVideo pointing to the magnified output video.

Source code in musicalgestures/_eulerian.py

def mg_eulerian(
        self: "musicalgestures.MgVideo",
        mode='color',
        freq_low=0.83,
        freq_high=1.0,
        amplification=50,
        levels=4,
        chroma_attenuation=1.0,
        lambda_cutoff=16,
        target_name=None,
        overwrite=True) -> "musicalgestures.MgVideo":
    """
    Applies Eulerian Video Magnification (EVM) to reveal subtle changes in a video.

    EVM amplifies small temporal variations that are normally invisible. Two modes are
    available:

    * ``mode='color'`` — amplifies subtle **colour** changes (e.g. blood flow / pulse,
      breathing). Uses a Gaussian pyramid and an ideal (FFT) temporal band-pass filter.
      Processed in two passes so only a small down-sampled stack is held in memory.
    * ``mode='motion'`` — amplifies subtle **motion**. Uses a Laplacian pyramid with a
      streaming IIR temporal band-pass filter and spatial-wavelength attenuation, so it
      runs frame-by-frame with low memory use.

    Based on Wu et al., "Eulerian Video Magnification for Revealing Subtle Changes in the
    World" (SIGGRAPH 2012).

    Args:
        mode (str, optional): 'color' or 'motion'. Defaults to 'color'.
        freq_low (float, optional): Lower temporal cutoff in Hz. Defaults to 0.83 (~50 bpm).
        freq_high (float, optional): Upper temporal cutoff in Hz. Defaults to 1.0 (~60 bpm).
        amplification (float, optional): Amplification factor (alpha). Defaults to 50.
        levels (int, optional): Number of spatial pyramid levels. Defaults to 4.
        chroma_attenuation (float, optional): Chrominance attenuation in [0, 1] (color mode).
            Lower values reduce colour artefacts. Defaults to 1.0.
        lambda_cutoff (float, optional): Spatial wavelength cutoff for amplitude attenuation
            (motion mode). Defaults to 16.
        target_name (str, optional): Target output name. Defaults to None (input filename with
            the suffix "_evm").
        overwrite (bool, optional): Whether to allow overwriting or auto-increment the filename.
            Defaults to True.

    Returns:
        MgVideo: An MgVideo pointing to the magnified output video.
    """
    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_evm' + fex, target_name, overwrite)

    mode = mode.lower()
    width, height, fps = self.width, self.height, self.fps
    # NB: for MgVideo, self.length is the frame count, not seconds.
    n_frames = self.length
    duration_s = self.length / fps if fps else 0

    def _open_reader():
        cmd = ['ffmpeg', '-y', '-i', self.filename]
        return ffmpeg_cmd(cmd, total_time=duration_s, pipe='read')

    def _open_writer():
        cmd = ['ffmpeg', '-y', '-s', f'{width}x{height}', '-r', str(fps), '-f', 'rawvideo',
               '-pix_fmt', 'bgr24', '-vcodec', 'rawvideo', '-i', '-',
               '-vcodec', 'libx264', '-pix_fmt', 'yuv420p', target_name]
        return ffmpeg_cmd(cmd, total_time=duration_s, pipe='write')

    frame_bytes = width * height * 3

    if mode == 'color':
        # ---- Pass 1: collect the small Gaussian level for every frame ----
        pb = MgProgressbar(total=n_frames * 2, prefix='EVM (color):')
        process = _open_reader()
        small_stack = []
        i = 0
        while True:
            buf = process.stdout.read(frame_bytes)
            if len(buf) < frame_bytes:
                break
            frame = np.frombuffer(buf, dtype=np.uint8).reshape(height, width, 3).astype(np.float32)
            small_stack.append(_gaussian_top(frame, levels))
            i += 1
            pb.progress(i)

        if len(small_stack) < 2:
            raise RuntimeError(f"Not enough frames in {self.filename} for EVM.")

        stack = np.stack(small_stack, axis=0)
        del small_stack

        # Temporal band-pass + amplification
        filtered = _ideal_bandpass(stack, fps, freq_low, freq_high)
        filtered[..., 0] *= amplification                       # B
        filtered[..., 1] *= amplification * chroma_attenuation  # G
        filtered[..., 2] *= amplification * chroma_attenuation  # R
        del stack

        # ---- Pass 2: add upsampled magnified signal back, write out ----
        process = _open_reader()
        writer = _open_writer()
        i = 0
        while True:
            buf = process.stdout.read(frame_bytes)
            if len(buf) < frame_bytes:
                break
            frame = np.frombuffer(buf, dtype=np.uint8).reshape(height, width, 3).astype(np.float32)
            mag = cv2.resize(filtered[i], (width, height))
            out = np.clip(frame + mag, 0, 255).astype(np.uint8)
            writer.stdin.write(out.tobytes())
            i += 1
            pb.progress(n_frames + i)
        writer.stdin.close()
        writer.wait()
        pb.progress(n_frames * 2)

    elif mode == 'motion':
        # ---- Streaming Laplacian-pyramid IIR band-pass ----
        pb = MgProgressbar(total=n_frames, prefix='EVM (motion):')
        process = _open_reader()
        writer = _open_writer()

        lowpass1: "list | None" = None
        lowpass2: "list | None" = None
        # IIR cutoffs derived from the requested band
        r1 = freq_high * 2.0 / fps
        r2 = freq_low * 2.0 / fps
        exaggeration = 2.0
        i = 0
        while True:
            buf = process.stdout.read(frame_bytes)
            if len(buf) < frame_bytes:
                break
            frame = np.frombuffer(buf, dtype=np.uint8).reshape(height, width, 3).astype(np.float32)
            pyr = _laplacian_pyramid(frame, levels)

            if lowpass1 is None:
                lowpass1 = [lvl.copy() for lvl in pyr]
                lowpass2 = [lvl.copy() for lvl in pyr]
                filtered = [np.zeros_like(lvl) for lvl in pyr]
            else:
                # built together on the first frame, above, and used together here
                assert lowpass1 is not None and lowpass2 is not None
                for k in range(len(pyr)):
                    lowpass1[k] = (1 - r1) * lowpass1[k] + r1 * pyr[k]
                    lowpass2[k] = (1 - r2) * lowpass2[k] + r2 * pyr[k]
                    filtered[k] = lowpass1[k] - lowpass2[k]

            # Spatial-wavelength dependent amplification (attenuate top & bottom levels)
            delta = lambda_cutoff / 8.0 / (1.0 + amplification)
            lambda_repr = (width ** 2 + height ** 2) ** 0.5 / 3.0
            amplified = []
            for k in range(len(pyr)):
                if k == 0 or k == len(pyr) - 1:
                    amplified.append(np.zeros_like(pyr[k]))
                else:
                    curr_alpha = lambda_repr / delta / 8.0 - 1.0
                    curr_alpha *= exaggeration
                    alpha_k = min(amplification, max(0.0, curr_alpha))
                    amplified.append(pyr[k] + filtered[k] * alpha_k)
                lambda_repr /= 2.0
            # Keep the residual (bottom) and finest (top) unamplified for stability
            amplified[-1] = pyr[-1]
            amplified[0] = pyr[0]

            out = _reconstruct_from_laplacian(amplified)
            out = np.clip(out, 0, 255).astype(np.uint8)
            writer.stdin.write(out.tobytes())
            i += 1
            pb.progress(i)

        writer.stdin.close()
        writer.wait()
        pb.progress(n_frames)

    else:
        raise ValueError(f"Unknown mode '{mode}'. Use 'color' or 'motion'.")

    self.eulerian_video = musicalgestures.MgVideo(target_name, color=self.color, returned_by_process=True)
    return self.eulerian_video

sonomotiongram

sonomotiongram(sonogram='vertical', n_fft=2048, sr=22050, n_iter=32, flip=True, normalize=True, target_name=None, overwrite=True)

Creates a sonomotiongram: a sonification of the video's motiongram.

The motiongram (a time–space image of where motion happens) is treated as a magnitude spectrogram — spatial position maps to frequency, motion intensity to amplitude — and converted back to audio with an inverse STFT (Griffin–Lim phase estimation). The result lets you hear the motion. Based on Jensenius, "Some video abstraction techniques for displaying body movement in analysis and performance" / sonomotiongrams (SMC 2013).

Parameters:

Name	Type	Description	Default
sonogram	str	Which motiongram to sonify: 'vertical' (motion across the vertical axis) or 'horizontal'. Defaults to 'vertical'.	'vertical'
n_fft	int	FFT size; sets the number of frequency bins (n_fft//2+1) the motiongram rows are mapped onto. Defaults to 2048.	2048
sr	int	Sample rate of the rendered audio. Defaults to 22050.	22050
n_iter	int	Griffin–Lim iterations for phase estimation (higher = cleaner, slower). Defaults to 32.	32
flip	bool	If True, map the top of the image to high frequencies (usually more intuitive). Defaults to True.	True
normalize	bool	Normalise the rendered audio to peak 1.0. Defaults to True.	True
target_name	str	Output audio filename. Defaults to None (input filename with the suffix "sono.wav").	None
overwrite	bool	Whether to allow overwriting or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgAudio		An MgAudio pointing to the rendered sonification (WAV).

Source code in musicalgestures/_sonification.py

def mg_sonomotiongram(
        self,
        sonogram='vertical',
        n_fft=2048,
        sr=22050,
        n_iter=32,
        flip=True,
        normalize=True,
        target_name=None,
        overwrite=True):
    """
    Creates a *sonomotiongram*: a sonification of the video's motiongram.

    The motiongram (a time–space image of where motion happens) is treated as a magnitude
    spectrogram — spatial position maps to frequency, motion intensity to amplitude — and
    converted back to audio with an inverse STFT (Griffin–Lim phase estimation). The result
    lets you *hear* the motion. Based on Jensenius, "Some video abstraction techniques for
    displaying body movement in analysis and performance" / sonomotiongrams (SMC 2013).

    Args:
        sonogram (str, optional): Which motiongram to sonify: 'vertical' (motion across the
            vertical axis) or 'horizontal'. Defaults to 'vertical'.
        n_fft (int, optional): FFT size; sets the number of frequency bins (n_fft//2+1) the
            motiongram rows are mapped onto. Defaults to 2048.
        sr (int, optional): Sample rate of the rendered audio. Defaults to 22050.
        n_iter (int, optional): Griffin–Lim iterations for phase estimation (higher = cleaner,
            slower). Defaults to 32.
        flip (bool, optional): If True, map the top of the image to high frequencies (usually
            more intuitive). Defaults to True.
        normalize (bool, optional): Normalise the rendered audio to peak 1.0. Defaults to True.
        target_name (str, optional): Output audio filename. Defaults to None (input filename
            with the suffix "_sono_<sonogram>.wav").
        overwrite (bool, optional): Whether to allow overwriting or auto-increment the filename.
            Defaults to True.

    Returns:
        MgAudio: An MgAudio pointing to the rendered sonification (WAV).
    """
    import librosa

    sonogram = sonogram.lower()
    if sonogram not in ('vertical', 'horizontal'):
        raise ValueError("sonogram must be 'vertical' or 'horizontal'.")

    target_name = resolve_filename(self.of, f"_sono_{sonogram}.wav", target_name, overwrite)

    width, height, fps = self.width, self.height, self.fps
    # NB: for MgVideo, self.length is the frame count, not seconds.
    duration_s = self.length / fps if fps else 0
    frame_bytes = width * height * 3

    # --- Build the motiongram (magnitude over time) from frame differences ---
    cmd = ['ffmpeg', '-y', '-i', self.filename]
    process = ffmpeg_cmd(cmd, total_time=duration_s, pipe='read')
    pb = MgProgressbar(total=self.length, prefix='Building motiongram for sonification:')

    columns = []
    prev_gray = None
    n = 0
    while True:
        buf = process.stdout.read(frame_bytes)
        if len(buf) < frame_bytes:
            break
        frame = np.frombuffer(buf, dtype=np.uint8).reshape(height, width, 3)
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
        if prev_gray is not None:
            motion = np.abs(gray - prev_gray)
            if sonogram == 'vertical':
                columns.append(motion.mean(axis=1))   # length H (vertical position)
            else:
                columns.append(motion.mean(axis=0))   # length W (horizontal position)
        prev_gray = gray
        n += 1
        pb.progress(n)
    pb.progress(self.length)

    if len(columns) < 2:
        raise RuntimeError(f"Not enough frames in {self.filename} to build a sonomotiongram.")

    gram = np.stack(columns, axis=1)  # shape (space, time)
    if flip:
        gram = gram[::-1, :]

    # --- Treat the motiongram as a magnitude spectrogram and invert it ---
    n_freq = n_fft // 2 + 1
    n_time = gram.shape[1]
    # Map spatial rows -> frequency bins
    mag = cv2.resize(gram.astype(np.float32), (n_time, n_freq), interpolation=cv2.INTER_LINEAR)

    # Scale magnitudes to a useful range
    if mag.max() > 0:
        mag = mag / mag.max()

    # Choose hop so the audio duration matches the video duration
    total_samples = int(sr * duration_s)
    hop_length = max(1, total_samples // n_time)

    y = librosa.griffinlim(mag, n_iter=n_iter, hop_length=hop_length, win_length=n_fft, n_fft=n_fft)

    if normalize:
        peak = np.max(np.abs(y))
        if peak > 0:
            y = y / peak

    # Write the WAV
    try:
        import soundfile as sf
        sf.write(target_name, y.astype(np.float32), sr)
    except Exception:
        from scipy.io import wavfile
        wavfile.write(target_name, sr, (y * 32767).astype(np.int16))

    # Save the result as sonomotiongram_audio for the parent MgVideo. NB: not
    # `self.sonomotiongram`, which would overwrite (shadow) the bound method
    # and break any subsequent sonomotiongram() call on the same object.
    self.sonomotiongram_audio = musicalgestures.MgAudio(target_name, sr=sr)
    return self.sonomotiongram_audio

stroboscope

stroboscope(n_samples=12, method='auto', threshold=0.1, kernel_size=5, keep_largest=False, colorize=True, background='average', target_name=None, overwrite=True)

Deprecated since 1.23.0: use multishot(), which absorbed this.

The two were the same picture made two ways, and having both meant a reader had to know which. multishot() now carries both: select='even' is this method's regular sampling, background='average' is its mean-average ground, and colorize=True is its time tint. The defaults differ because the defaults are an opinion --- moments chosen for spatial separation, on a median plate --- and even sampling onto a mean average is what makes two bodies land in the same place on a ground that keeps a ghost of everyone who crossed.

This wrapper delegates and will be removed at 2.0.

Parameters:

Name	Type	Description	Default
n_samples	int	Silhouettes to composite. Defaults to 12.	12
method	str	'auto', 'mediapipe' or 'bgsub'. Defaults to 'auto'.	'auto'
threshold	float	Foreground threshold (0-1). Defaults to 0.1.	0.1
kernel_size	int	Unused by the replacement; accepted for compatibility.	5
keep_largest	bool	Unused by the replacement; accepted for compatibility.	False
colorize	bool	Tint each silhouette by time. Defaults to True.	True
background	str	'average', 'first', 'black' or 'white'. Defaults to 'average'.	'average'
target_name	str	Output name. Defaults to "_stroboscope.png".	None
overwrite	bool	Overwrite or auto-increment. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	the composite.

Source code in musicalgestures/_spacetime.py

def mg_stroboscope(self: "musicalgestures.MgVideo", n_samples: int = 12, method: str = 'auto', threshold: float = 0.1, kernel_size: int = 5,
                   keep_largest: bool = False, colorize: bool = True, background: str = 'average',
                   target_name: str | None = None, overwrite: bool = True) -> "MgImage":
    """
    **Deprecated since 1.23.0: use** ``multishot()``, **which absorbed this.**

    The two were the same picture made two ways, and having both meant a reader had to
    know which. `multishot()` now carries both: ``select='even'`` is this method's regular
    sampling, ``background='average'`` is its mean-average ground, and ``colorize=True``
    is its time tint. The defaults differ because the defaults are an opinion --- moments
    chosen for spatial separation, on a median plate --- and even sampling onto a mean
    average is what makes two bodies land in the same place on a ground that keeps a ghost
    of everyone who crossed.

    This wrapper delegates and will be removed at 2.0.

    Args:
        n_samples (int, optional): Silhouettes to composite. Defaults to 12.
        method (str, optional): 'auto', 'mediapipe' or 'bgsub'. Defaults to 'auto'.
        threshold (float, optional): Foreground threshold (0-1). Defaults to 0.1.
        kernel_size (int, optional): Unused by the replacement; accepted for compatibility.
        keep_largest (bool, optional): Unused by the replacement; accepted for compatibility.
        colorize (bool, optional): Tint each silhouette by time. Defaults to True.
        background (str, optional): 'average', 'first', 'black' or 'white'. Defaults to 'average'.
        target_name (str, optional): Output name. Defaults to "_stroboscope.png".
        overwrite (bool, optional): Overwrite or auto-increment. Defaults to True.

    Returns:
        MgImage: the composite.
    """
    warnings.warn(
        "stroboscope() is deprecated and will be removed at 2.0; use multishot("
        "select='even', background='average', colorize=True), which absorbed it.",
        DeprecationWarning, stacklevel=2)

    from musicalgestures._multishot import mg_multishot
    from musicalgestures._utils import resolve_filename

    target_name = resolve_filename(self.of, '_stroboscope.png', target_name, overwrite)
    return mg_multishot(
        self, n_bodies=n_samples, select="even", background=background,
        colorize=colorize, segmenter="plate" if method == "bgsub" else "auto",
        target_name=target_name, overwrite=overwrite)

silhouette_waterfall

silhouette_waterfall(n_samples=40, method='auto', threshold=0.1, kernel_size=5, keep_largest=False, axis='horizontal', cmap='viridis', dpi=200, elev=35, azim=-60, axes=True, crop=False, target_name=None, overwrite=True)

Renders a 3D silhouette waterfall: the per-frame silhouette projected onto one spatial axis and stacked as cascading curves along a time (depth) axis, so the body's occupancy profile "flows" through time — like a 3D spectrogram waterfall.

For a single person on a static background, raise threshold and/or set keep_largest=True for a cleaner profile.

Parameters:

Name	Type	Description	Default
n_samples	int	Number of time slices (profiles) to stack. Defaults to 40.	40
method	str	Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.	'auto'
threshold	float	Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.	0.1
kernel_size	int	Morphological cleanup kernel (0 disables). Defaults to 5.	5
keep_largest	bool	Keep only the largest blob (the person). Defaults to False.	False
axis	str	'horizontal' profiles over x (collapse y); 'vertical' profiles over y. Defaults to 'horizontal'.	'horizontal'
cmap	str	Matplotlib colormap (by time). Defaults to 'viridis'.	'viridis'
dpi	int	Output DPI. Defaults to 200.	200
elev	float	3D elevation angle. Defaults to 35.	35
azim	float	3D azimuth angle. Defaults to -60.	-60
axes	bool	Draw the axes, tick labels, and title. Set to False for a clean render with all axes and text removed. Defaults to True.	True
crop	bool	Tighten the spatial axis to the occupied (nonzero) extent and trim the surrounding whitespace, so the figure shows mostly the data. Defaults to False.	False
target_name	str	Output name. Defaults to None ("_silhouette_waterfall.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	the 3D waterfall figure (the stacked profiles are in .data).

Source code in musicalgestures/_spacetime.py

def mg_silhouette_waterfall(self: "musicalgestures.MgVideo", n_samples: int = 40, method: str = 'auto', threshold: float = 0.1, kernel_size: int = 5,
                            keep_largest: bool = False, axis: str = 'horizontal', cmap: str = 'viridis', dpi: int = 200,
                            elev: float = 35, azim: float = -60, axes: bool = True, crop: bool = False, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Renders a 3D silhouette waterfall: the per-frame silhouette projected onto one spatial
    axis and stacked as cascading curves along a time (depth) axis, so the body's occupancy
    profile "flows" through time — like a 3D spectrogram waterfall.

    For a single person on a static background, raise ``threshold`` and/or set
    ``keep_largest=True`` for a cleaner profile.

    Args:
        n_samples (int, optional): Number of time slices (profiles) to stack. Defaults to 40.
        method (str, optional): Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.
        threshold (float, optional): Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.
        kernel_size (int, optional): Morphological cleanup kernel (0 disables). Defaults to 5.
        keep_largest (bool, optional): Keep only the largest blob (the person). Defaults to False.
        axis (str, optional): 'horizontal' profiles over x (collapse y); 'vertical' profiles over y. Defaults to 'horizontal'.
        cmap (str, optional): Matplotlib colormap (by time). Defaults to 'viridis'.
        dpi (int, optional): Output DPI. Defaults to 200.
        elev (float, optional): 3D elevation angle. Defaults to 35.
        azim (float, optional): 3D azimuth angle. Defaults to -60.
        axes (bool, optional): Draw the axes, tick labels, and title. Set to False for a clean
            render with all axes and text removed. Defaults to True.
        crop (bool, optional): Tighten the spatial axis to the occupied (nonzero) extent and trim
            the surrounding whitespace, so the figure shows mostly the data. Defaults to False.
        target_name (str, optional): Output name. Defaults to None ("_silhouette_waterfall.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgFigure: the 3D waterfall figure (the stacked profiles are in ``.data``).
    """
    import matplotlib
    import matplotlib.pyplot as plt
    from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 (registers 3d projection)

    target_name = resolve_filename(self.of, '_silhouette_waterfall.png', target_name, overwrite)

    avg = _average_frame(self)
    bg_gray = cv2.cvtColor(avg, cv2.COLOR_BGR2GRAY).astype(np.float32)
    seg_fn = _make_segmenter(method)

    total = int(self.length)
    sample_idx = sorted(set(np.linspace(0, total - 1, min(n_samples, total)).astype(int).tolist()))
    sample_set = set(sample_idx)

    profiles = []
    times = []
    pb = MgProgressbar(total=self.length, prefix='Rendering silhouette waterfall:')
    i = 0
    for frame in _iter_frames(self):
        if i in sample_set:
            mask = _silhouette(frame, seg_fn, bg_gray, threshold, kernel_size, keep_largest)
            if axis == 'horizontal':
                profiles.append(mask.sum(axis=0).astype(np.float32))   # over x (length W)
            else:
                profiles.append(mask.sum(axis=1).astype(np.float32))   # over y (length H)
            times.append(i / max(self.fps, 1))
        i += 1
        pb.progress(i)
    pb.progress(self.length)

    arr = np.array(profiles, dtype=np.float32)  # (n_slices, axis_len)
    if arr.size and arr.max() > 0:
        arr = arr / arr.max()

    cmap_obj = matplotlib.colormaps[cmap]
    fig = plt.figure(figsize=(11, 8), dpi=dpi)
    ax = fig.add_subplot(111, projection='3d')
    fig.patch.set_facecolor('white')

    pos = np.arange(arr.shape[1]) if arr.size else np.array([])
    n_slices = max(len(profiles) - 1, 1)
    for k, (prof, t) in enumerate(zip(arr, times)):
        ax.plot(pos, np.full_like(pos, t, dtype=float), prof,
                color=cmap_obj(k / n_slices), lw=0.9, alpha=0.9)

    if crop and arr.size:
        # Tighten the spatial axis to the occupied (nonzero) profile extent.
        occupied = np.where(arr.max(axis=0) > 0)[0]
        if occupied.size:
            pad = max(int((occupied[-1] - occupied[0]) * 0.05), 1)
            ax.set_xlim(max(occupied[0] - pad, 0), min(occupied[-1] + pad, arr.shape[1] - 1))

    if axes:
        ax.set_xlabel('Horizontal position (px)' if axis == 'horizontal' else 'Vertical position (px)')
        ax.set_ylabel('Time (s)')
        ax.set_zlabel('Silhouette extent')
        ax.set_title('Silhouette waterfall')
    else:
        ax.set_axis_off()
    ax.view_init(elev=elev, azim=azim)
    if crop:
        ax.set_position([0, 0, 1, 1])
        try:
            ax.set_box_aspect(None, zoom=1.5)
        except TypeError:
            pass
        save_kwargs = {'bbox_inches': 'tight', 'pad_inches': 0}
    else:
        fig.tight_layout()
        save_kwargs = {}
    fig.savefig(target_name, facecolor='white', **save_kwargs)
    plt.close(fig)

    data = {'profiles': arr, 'times': np.array(times), 'axis': axis}
    mgf = MgFigure(figure=fig, figure_type='video.silhouette_waterfall', data=data, layers=None, image=target_name)
    self.silhouette_waterfall_figure = mgf
    return mgf

motionhistory

motionhistory(threshold=0.05, decay=0.3, normalize=False, blur=0, cmap='hot', dpi=300, target_name=None, overwrite=True)

Renders a Motion History Image (Bobick & Davis): a single image where intensity encodes how recently motion occurred at each pixel (recent motion bright, older motion fades out).

A motion mark is set to full intensity where motion occurs and then decays linearly to zero over a window set by decay, so old motion disappears instead of accumulating and washing out the image. Raise threshold to ignore background noise, and lower decay for shorter (less crowded) trails.

Parameters:

Name	Type	Description	Default
threshold	float	Motion threshold (0–1) on frame differences. Higher rejects more background noise. Defaults to 0.05.	0.05
decay	float	Fade window as a fraction of the clip length (0–1): a motion mark fully fades after this fraction of the video. Smaller = shorter trails, less blow-out. Defaults to 0.3.	0.3
normalize	bool	Stretch the result to the full intensity range. Defaults to False. The MHI is already built in [0, 1], so normalization is rarely needed; when the final frames are static it amplifies faint residual trails and over-brightens ("blows up") the image, so it is guarded to skip when the peak intensity is very low.	False
blur	int	Optional Gaussian smoothing radius for the difference mask (0 = off). Helps suppress speckle noise. Defaults to 0.	0
cmap	str	Matplotlib colormap. Defaults to 'hot'.	'hot'
dpi	int	Output DPI. Defaults to 300.	300
target_name	str	Output name. Defaults to None ("_mhi.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	the motion history image.

Source code in musicalgestures/_spacetime.py

def mg_motionhistory(self: "musicalgestures.MgVideo", threshold: float = 0.05, decay: float = 0.3, normalize: bool = False, blur: int = 0,
                     cmap: str = 'hot', dpi: int = 300, target_name: str | None = None, overwrite: bool = True) -> "MgImage":
    """
    Renders a Motion History Image (Bobick & Davis): a single image where intensity encodes
    how recently motion occurred at each pixel (recent motion bright, older motion fades out).

    A motion mark is set to full intensity where motion occurs and then **decays** linearly to
    zero over a window set by ``decay``, so old motion disappears instead of accumulating and
    washing out the image. Raise ``threshold`` to ignore background noise, and lower ``decay``
    for shorter (less crowded) trails.

    Args:
        threshold (float, optional): Motion threshold (0–1) on frame differences. Higher rejects
            more background noise. Defaults to 0.05.
        decay (float, optional): Fade window as a fraction of the clip length (0–1): a motion
            mark fully fades after this fraction of the video. Smaller = shorter trails, less
            blow-out. Defaults to 0.3.
        normalize (bool, optional): Stretch the result to the full intensity range. Defaults to False.
            The MHI is already built in [0, 1], so normalization is rarely needed; when the final
            frames are static it amplifies faint residual trails and over-brightens ("blows up") the
            image, so it is guarded to skip when the peak intensity is very low.
        blur (int, optional): Optional Gaussian smoothing radius for the difference mask (0 = off).
            Helps suppress speckle noise. Defaults to 0.
        cmap (str, optional): Matplotlib colormap. Defaults to 'hot'.
        dpi (int, optional): Output DPI. Defaults to 300.
        target_name (str, optional): Output name. Defaults to None ("_mhi.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgImage: the motion history image.
    """
    import matplotlib
    import matplotlib.pyplot as plt

    target_name = resolve_filename(self.of, '_mhi.png', target_name, overwrite)

    total = max(int(self.length), 1)
    decay_frames = max(1, int(decay * total))
    step = 1.0 / decay_frames

    mhi = np.zeros((self.height, self.width), dtype=np.float32)
    prev_gray = None
    pb = MgProgressbar(total=self.length, prefix='Rendering motion history image:')
    i = 0
    for frame in _iter_frames(self):
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
        if prev_gray is not None:
            diff = np.abs(gray - prev_gray)
            if blur and blur > 0:
                k = int(blur) * 2 + 1
                diff = cv2.GaussianBlur(diff, (k, k), 0)
            motion = diff > (threshold * 255)
            # Decay everything, then re-stamp current motion to full intensity
            mhi -= step
            np.clip(mhi, 0.0, 1.0, out=mhi)
            mhi[motion] = 1.0
        prev_gray = gray
        i += 1
        pb.progress(i)
    pb.progress(self.length)

    # Only normalize when there is substantial motion intensity; otherwise dividing by a tiny
    # peak amplifies faint residual trails into a washed-out ("blown up") image.
    if normalize and mhi.max() > 0.2:
        mhi = mhi / mhi.max()

    fig, ax = plt.subplots(figsize=(12, 12 * self.height / self.width), dpi=dpi)
    fig.patch.set_facecolor('white')
    ax.imshow(mhi, cmap=cmap, vmin=0.0, vmax=1.0)
    ax.set_title('Motion History Image (bright = recent motion)')
    ax.axis('off')
    fig.tight_layout()
    fig.savefig(target_name, facecolor='white', bbox_inches='tight')
    plt.close(fig)

    self.mhi_image = MgImage(target_name)
    return self.mhi_image

spacetime_volume

spacetime_volume(n_samples=50, downsample=8, method='auto', threshold=0.1, kernel_size=5, keep_largest=False, cmap='viridis', dpi=200, elev=20, azim=-60, target_name=None, overwrite=True)

Renders a 3D space-time scatter of the person's silhouette: points (x, y, t) where the silhouette is present, with time on the depth axis and colour, showing how the body occupies space through time.

Parameters:

Name	Type	Description	Default
n_samples	int	Number of time samples (depth slices). Defaults to 50.	50
downsample	int	Spatial downsampling factor for the silhouette points. Defaults to 8.	8
method	str	Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.	'auto'
threshold	float	Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.	0.1
kernel_size	int	Morphological cleanup kernel for the silhouette (0 disables). Defaults to 5.	5
keep_largest	bool	Keep only the largest blob (the person). Defaults to False.	False
cmap	str	Matplotlib colormap for time. Defaults to 'viridis'.	'viridis'
dpi	int	Output DPI. Defaults to 200.	200
elev	float	3D elevation angle. Defaults to 20.	20
azim	float	3D azimuth angle. Defaults to -60.	-60
target_name	str	Output name. Defaults to None ("_spacetime_volume.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	the 3D space-time figure (data holds the point cloud).

Source code in musicalgestures/_spacetime.py

def mg_spacetime_volume(self: "musicalgestures.MgVideo", n_samples: int = 50, downsample: int = 8, method: str = 'auto', threshold: float = 0.1,
                        kernel_size: int = 5, keep_largest: bool = False, cmap: str = 'viridis', dpi: int = 200,
                        elev: float = 20, azim: float = -60, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Renders a 3D space-time scatter of the person's silhouette: points (x, y, t) where the
    silhouette is present, with time on the depth axis and colour, showing how the body
    occupies space through time.

    Args:
        n_samples (int, optional): Number of time samples (depth slices). Defaults to 50.
        downsample (int, optional): Spatial downsampling factor for the silhouette points. Defaults to 8.
        method (str, optional): Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.
        threshold (float, optional): Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.
        kernel_size (int, optional): Morphological cleanup kernel for the silhouette (0 disables). Defaults to 5.
        keep_largest (bool, optional): Keep only the largest blob (the person). Defaults to False.
        cmap (str, optional): Matplotlib colormap for time. Defaults to 'viridis'.
        dpi (int, optional): Output DPI. Defaults to 200.
        elev (float, optional): 3D elevation angle. Defaults to 20.
        azim (float, optional): 3D azimuth angle. Defaults to -60.
        target_name (str, optional): Output name. Defaults to None ("_spacetime_volume.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgFigure: the 3D space-time figure (data holds the point cloud).
    """
    import matplotlib
    import matplotlib.pyplot as plt
    from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 (registers 3d projection)

    target_name = resolve_filename(self.of, '_spacetime_volume.png', target_name, overwrite)

    avg = _average_frame(self)
    bg_gray = cv2.cvtColor(avg, cv2.COLOR_BGR2GRAY).astype(np.float32)
    seg_fn = _make_segmenter(method)

    total = int(self.length)
    sample_idx = set(np.linspace(0, total - 1, min(n_samples, total)).astype(int).tolist())

    xs, ys, ts = [], [], []
    pb = MgProgressbar(total=self.length, prefix='Building space-time volume:')
    i = 0
    for frame in _iter_frames(self):
        if i in sample_idx:
            mask = _silhouette(frame, seg_fn, bg_gray, threshold, kernel_size, keep_largest)
            sub = mask[::downsample, ::downsample]
            yy, xx = np.nonzero(sub)
            xs.append(xx * downsample)
            ys.append(yy * downsample)
            ts.append(np.full(len(xx), i / max(self.fps, 1)))  # seconds
        i += 1
        pb.progress(i)
    pb.progress(self.length)

    xs = np.concatenate(xs) if xs else np.array([])
    ys = np.concatenate(ys) if ys else np.array([])
    ts = np.concatenate(ts) if ts else np.array([])

    fig = plt.figure(figsize=(10, 8), dpi=dpi)
    ax = fig.add_subplot(111, projection='3d')
    fig.patch.set_facecolor('white')
    if len(xs):
        ax.scatter(xs, ts, self.height - ys, c=ts, cmap=cmap, s=2, alpha=0.5, depthshade=True)
    ax.set_xlabel('x (px)')
    ax.set_ylabel('time (s)')
    ax.set_zlabel('y (px)')
    ax.set_title('Space-time silhouette volume')
    ax.view_init(elev=elev, azim=azim)
    fig.tight_layout()
    fig.savefig(target_name, facecolor='white')
    plt.close(fig)

    data = {'x': xs, 'y': ys, 't': ts, 'fps': self.fps}
    mgf = MgFigure(figure=fig, figure_type='video.spacetime_volume', data=data, layers=None, image=target_name)
    self.spacetime_volume_figure = mgf
    return mgf

beat_statistics

beat_statistics(source='motion', n_bins=32, cmap='YlOrRd', dpi=300, autoshow=True, title=None, target_name=None, overwrite=True, fmin=0.2, fmax=8.0)

Circular statistics of beat-timing consistency, from the audio or from the motion.

Fits an ideal isochronous beat grid to the detected beats and visualises how each beat deviates from it (a polar phase histogram with the mean resultant vector, plus a millisecond-deviation time series), revealing whether the timing rushes, drags, or stays steady. Requires at least four detected beats.

Parameters:

Name	Type	Description	Default
source	str	'motion' (default) detects onsets in the video's quantity of motion and analyses their timing; 'audio' analyses the audio track instead (same as MgAudio.beat_statistics / video.audio.beat_statistics).	'motion'
n_bins	int	Bins in the polar phase histogram. Defaults to 32.	32
cmap	str	Colormap for the polar histogram. Defaults to 'YlOrRd'.	'YlOrRd'
dpi	int	Output DPI. Defaults to 300.	300
autoshow	bool	Kept for API parity (display is via show()). Defaults to True.	True
title	str	Optional figure title; use 'filename' for the file name. Defaults to None.	None
target_name	str	Output image name. Defaults to None.	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
fmin	float	Lowest motion-onset rate to consider (Hz), 'motion' only. Defaults to 0.2.	0.2
fmax	float	Highest motion-onset rate to consider (Hz), 'motion' only. Defaults to 8.0.	8.0

Returns:

Name	Type	Description
MgFigure	'MgFigure | None'	figure with the beat statistics in .data, or None if too few beats.

Source code in musicalgestures/_movementbeats.py

def mg_beat_statistics(self, source: str = 'motion', n_bins: int = 32, cmap: str = 'YlOrRd', dpi: int = 300,
                       autoshow: bool = True, title: str | None = None, target_name: str | None = None, overwrite: bool = True,
                       fmin: float = 0.2, fmax: float = 8.0) -> "MgFigure | None":
    """
    Circular statistics of beat-timing consistency, from the **audio** or from the **motion**.

    Fits an ideal isochronous beat grid to the detected beats and visualises how each beat
    deviates from it (a polar phase histogram with the mean resultant vector, plus a
    millisecond-deviation time series), revealing whether the timing rushes, drags, or stays
    steady. Requires at least four detected beats.

    Args:
        source (str, optional): `'motion'` (default) detects onsets in the video's
            quantity of motion and analyses their timing; `'audio'` analyses the
            audio track instead (same as `MgAudio.beat_statistics` / `video.audio.beat_statistics`).
        n_bins (int, optional): Bins in the polar phase histogram. Defaults to 32.
        cmap (str, optional): Colormap for the polar histogram. Defaults to 'YlOrRd'.
        dpi (int, optional): Output DPI. Defaults to 300.
        autoshow (bool, optional): Kept for API parity (display is via show()). Defaults to True.
        title (str, optional): Optional figure title; use 'filename' for the file name. Defaults to None.
        target_name (str, optional): Output image name. Defaults to None.
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        fmin (float, optional): Lowest motion-onset rate to consider (Hz), 'motion' only. Defaults to 0.2.
        fmax (float, optional): Highest motion-onset rate to consider (Hz), 'motion' only. Defaults to 8.0.

    Returns:
        MgFigure: figure with the beat statistics in ``.data``, or None if too few beats.
    """
    source = str(source).lower()
    if source == 'audio':
        # Delegate to the inherited audio implementation (operates on the audio track)
        from musicalgestures._audio import MgAudio
        return MgAudio.beat_statistics(self, n_bins=n_bins, cmap=cmap, dpi=dpi,
                                       autoshow=autoshow, title=title,
                                       target_name=target_name, overwrite=overwrite)
    if source != 'motion':
        raise ValueError("source must be 'audio' or 'motion'.")

    from musicalgestures._analysis import circular_stats, rayleigh_test
    import librosa

    qom, fps = _movement_qom(self)
    if len(qom) < 8:
        print('Not enough frames to detect motion onsets.')
        return None

    # Detect motion onsets ("beats") as peaks in the quantity-of-motion envelope.
    beat_times = librosa.onset.onset_detect(
        onset_envelope=qom, sr=fps, hop_length=1, units='time', backtrack=False)
    # Keep onsets within the plausible motion-rate band via inter-onset interval
    if len(beat_times) >= 2:
        ibi_all = np.diff(beat_times)
        keep = (ibi_all >= 1.0 / fmax) & (ibi_all <= 1.0 / fmin)
        beat_times = np.concatenate([beat_times[:1], beat_times[1:][keep]])

    if len(beat_times) < 4:
        print('Not enough motion onsets detected for circular statistics (need at least 4).')
        return None

    # Circular grid statistics (same model as the audio version)
    k = np.arange(len(beat_times))
    T_fit, t0_fit = np.polyfit(k, beat_times, 1)
    deviations_s = beat_times - (t0_fit + k * T_fit)
    beat_phases = (deviations_s / T_fit) * 2 * np.pi % (2 * np.pi)
    R_beat, mu_beat = circular_stats(beat_phases)
    _, p_rayleigh = rayleigh_test(beat_phases)
    ibi = np.diff(beat_times)
    beat_regularity = float(1.0 - ibi.std() / ibi.mean()) if len(ibi) and ibi.mean() > 0 else 0.0
    tempo = 60.0 / T_fit if T_fit > 0 else 0.0

    d = {
        'source': 'motion',
        'fps': fps,
        'of': self.of,
        'tempo': tempo,
        'beat_times': beat_times,
        'ibi': ibi,
        'beat_regularity': beat_regularity,
        'beat_phases': beat_phases,
        'deviations_s': deviations_s,
        'R_beat': R_beat,
        'mu_beat': mu_beat,
        'T_fit': T_fit,
        't0_fit': t0_fit,
        'p_rayleigh': p_rayleigh,
        'qom': qom,
    }

    target_name = resolve_filename(self.of, '_movement_beatstats.png', target_name, overwrite)

    deviations_ms = deviations_s * 1000
    R, mu = R_beat, mu_beat

    fig = plt.figure(figsize=(14, 6), dpi=dpi)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    # Polar histogram of motion-beat phases
    ax_p = fig.add_subplot(121, projection='polar')
    bin_edges = np.linspace(0, 2 * np.pi, n_bins + 1)
    counts, _ = np.histogram(beat_phases, bins=bin_edges)
    theta_c = (bin_edges[:-1] + bin_edges[1:]) / 2
    norm_c = counts / (counts.max() + 1e-9)
    ax_p.bar(theta_c, counts, width=2 * np.pi / n_bins * 0.88,
             color=matplotlib.colormaps[cmap](norm_c), alpha=0.85,
             edgecolor='white', linewidth=0.3)
    if counts.max() > 0:
        ax_p.annotate('', xy=(np.radians(mu), R * counts.max()), xytext=(0, 0),
                      arrowprops=dict(arrowstyle='-|>', color='#333333', lw=2.0, mutation_scale=16))
    ax_p.set_xticks([0, np.pi / 2, np.pi, 3 * np.pi / 2])
    ax_p.set_xticklabels(['on beat', '1/4 late', '1/2', '1/4 early'], fontsize=8)
    ax_p.set_title(f'Motion-beat phase deviation\nR = {R:.3f}   μ = {mu:.1f}°   p = {p_rayleigh:.4f}', fontsize=10)

    # Time series of deviations
    ax_t = fig.add_subplot(122)
    sc = ax_t.scatter(beat_times, deviations_ms, c=beat_times, cmap='plasma', s=18, alpha=0.8)
    ax_t.axhline(0, color='#888888', lw=1.0, ls='--', alpha=0.7)
    ax_t.axhline(float(deviations_ms.mean()), color='#1f77b4', lw=1.2, ls=':',
                 label=f'mean {float(deviations_ms.mean()):.1f} ms')
    ax_t.set(xlabel='Time (s)', ylabel='Deviation from ideal grid (ms)', title='Motion-beat timing deviation')
    ax_t.legend()
    cb = fig.colorbar(sc, ax=ax_t)
    cb.set_label('Time (s)')

    if title is None:
        title = ''
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(f'{title}   Motion tempo: {tempo:.1f} BPM   σ = {float(deviations_ms.std()):.1f} ms'.strip(),
                 fontsize=13, fontweight='bold')

    plt.tight_layout(rect=[0, 0, 1, 0.93])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    mgf = MgFigure(figure=fig, figure_type='video.beat_statistics', data=d, layers=None, image=target_name)
    self.movement_beat_statistics_figure = mgf
    return mgf

tempo_similarity

tempo_similarity(dpi=300, autoshow=True, title=None, target_name=None, overwrite=True)

Compare the audio tempo with the motion tempo and report how similar they are.

Estimates the tempo of the audio track (from its onset-strength envelope) and of the motion (from the quantity-of-motion envelope), then aligns the two normalised envelopes and cross-correlates them to measure their agreement. The figure shows the two envelopes overlaid and their cross-correlation; the report (also saved as a CSV) lists the audio tempo, motion tempo, their ratio and nearest harmonic relationship, the peak cross-correlation, and the lag (s) at which the motion best aligns with the audio.

Parameters:

Name	Type	Description	Default
dpi	int	Output DPI. Defaults to 300.	300
autoshow	bool	Kept for API parity (display via show()). Defaults to True.	True
title	str	Optional figure title; 'filename' uses the file name. Defaults to None.	None
target_name	str	Output image name. Defaults to None ("_tempo_similarity.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure | None'	the report figure (metrics in .data), or None if the video has no audio.

Source code in musicalgestures/_movementbeats.py

def mg_tempo_similarity(self, dpi: int = 300, autoshow: bool = True, title: str | None = None, target_name: str | None = None, overwrite: bool = True) -> "MgFigure | None":
    """
    Compare the **audio** tempo with the **motion** tempo and report how similar they are.

    Estimates the tempo of the audio track (from its onset-strength envelope) and of the motion
    (from the quantity-of-motion envelope), then aligns the two normalised envelopes and
    cross-correlates them to measure their agreement. The figure shows the two envelopes
    overlaid and their cross-correlation; the report (also saved as a CSV) lists the audio tempo,
    motion tempo, their ratio and nearest harmonic relationship, the peak cross-correlation, and
    the lag (s) at which the motion best aligns with the audio.

    Args:
        dpi (int, optional): Output DPI. Defaults to 300.
        autoshow (bool, optional): Kept for API parity (display via show()). Defaults to True.
        title (str, optional): Optional figure title; 'filename' uses the file name. Defaults to None.
        target_name (str, optional): Output image name. Defaults to None ("_tempo_similarity.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgFigure: the report figure (metrics in ``.data``), or None if the video has no audio.
    """
    import librosa
    from musicalgestures._utils import has_audio

    if not has_audio(self.filename):
        print('The video has no audio track — cannot compare audio and motion tempo.')
        return None

    target_name = resolve_filename(self.of, '_tempo_similarity.png', target_name, overwrite)

    # --- Audio envelope + tempo ---
    y, sr = self._load()
    hop = self.hop_length
    oenv = librosa.onset.onset_strength(y=y, sr=sr, hop_length=hop)
    audio_tempo = float(np.atleast_1d(librosa.feature.tempo(onset_envelope=oenv, sr=sr, hop_length=hop))[0])
    a_t = librosa.times_like(oenv, sr=sr, hop_length=hop)

    # --- Motion envelope + tempo ---
    qom, fps = _movement_qom(self)
    if len(qom) < 4:
        print('Not enough frames to estimate motion tempo.')
        return None
    motion_tempo = float(np.atleast_1d(librosa.feature.tempo(onset_envelope=qom, sr=fps, hop_length=1))[0])
    m_t = np.arange(len(qom)) / max(fps, 1e-9)

    # --- Resample both onto a common time grid and cross-correlate ---
    fs = 50.0  # Hz
    dur = min(a_t[-1] if len(a_t) else 0, m_t[-1] if len(m_t) else 0)
    if dur <= 0:
        print('Audio and motion do not overlap in time.')
        return None
    t = np.arange(0, dur, 1.0 / fs)
    a = np.interp(t, a_t, oenv)
    m = np.interp(t, m_t, qom)
    a = (a - a.mean()) / (a.std() + 1e-9)
    m = (m - m.mean()) / (m.std() + 1e-9)
    xcorr = np.correlate(m, a, mode='full') / len(t)
    lags = np.arange(-len(t) + 1, len(t)) / fs
    peak_i = int(np.argmax(xcorr))
    peak_lag = float(lags[peak_i])
    peak_corr = float(xcorr[peak_i])
    zero_corr = float(xcorr[len(t) - 1])  # correlation at zero lag

    ratio = motion_tempo / audio_tempo if audio_tempo > 0 else 0.0
    harm, harm_label = _nearest_harmonic_ratio(ratio) if ratio > 0 else (0.0, 'n/a')
    tempo_agreement = float(max(0.0, 1.0 - abs(np.log2(ratio / harm)))) if (ratio > 0 and harm > 0) else 0.0

    # --- Figure ---
    fig, axes = plt.subplots(2, 1, figsize=(12, 7), dpi=dpi)
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Audio–motion tempo similarity', fontsize=14)

    axes[0].plot(t, a, color='#1f77b4', lw=0.9, label=f'Audio onset (tempo {audio_tempo:.1f} BPM)')
    axes[0].plot(t, m, color='#d62728', lw=0.9, alpha=0.8, label=f'Quantity of motion (tempo {motion_tempo:.1f} BPM)')
    axes[0].set_xlabel('Time (s)')
    axes[0].set_ylabel('Normalised envelope')
    axes[0].legend(loc='upper right', fontsize=8)
    axes[0].set_title('Audio onset strength vs. quantity of motion', fontsize=10)

    axes[1].plot(lags, xcorr, color='#2ca02c', lw=0.9)
    axes[1].axvline(peak_lag, color='crimson', ls='--', lw=1,
                    label=f'peak r={peak_corr:.2f} @ {peak_lag:+.2f}s')
    axes[1].axvline(0, color='#888888', ls=':', lw=0.8)
    axes[1].set_xlabel('Lag (s)  — motion relative to audio')
    axes[1].set_ylabel('Cross-correlation')
    axes[1].legend(loc='upper right', fontsize=8)
    axes[1].set_title(
        f'Tempo ratio {ratio:.2f} (≈ {harm_label})  |  agreement {tempo_agreement:.2f}  |  '
        f'zero-lag r={zero_corr:.2f}', fontsize=10)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    d = {
        'audio_tempo_bpm': round(audio_tempo, 2),
        'motion_tempo_bpm': round(motion_tempo, 2),
        'tempo_ratio': round(ratio, 3),
        'nearest_harmonic': harm_label,
        'tempo_agreement': round(tempo_agreement, 3),
        'peak_crosscorr': round(peak_corr, 3),
        'peak_lag_s': round(peak_lag, 3),
        'zero_lag_crosscorr': round(zero_corr, 3),
        'fps': fps, 'sr': sr,
    }
    try:
        import pandas as pd
        pd.DataFrame([d]).to_csv(os.path.splitext(target_name)[0] + '.csv', index=False)
    except Exception as e:
        print(f'Warning: could not save CSV: {e}')

    mgf = MgFigure(figure=fig, figure_type='video.tempo_similarity', data=d, layers=None, image=target_name)
    self.tempo_similarity_figure = mgf
    return mgf

motiondescriptors

motiondescriptors(window='hann', entropy_bins=50, fmin=0.2, fmax=10.0, save_data=True, save_plot=True, data_format='csv', target_name=None, overwrite=True)

Scalar motion descriptors derived from the quantity-of-motion (QoM) signal.

Computes a compact set of higher-level descriptors that summarise how something moves, complementing the per-frame motion data from :func:motion:

• motion_energy — mean squared QoM; the overall amount of motion.
• motion_smoothness — SPARC (spectral arc length) of the QoM profile; a dimensionless, validated smoothness metric (less negative = smoother, more negative = jerkier).
• motion_entropy — normalised (0–1) Shannon entropy of the QoM magnitude distribution; the complexity/variedness of the motion.
• spectral descriptors of the QoM signal (Hann-windowed by default): the dominant frequency (Hz, the dominant periodicity of the motion) and the spectral centroid (Hz, the "centre of mass" of the motion spectrum).

Parameters:

Name	Type	Description	Default
window	str	FFT window for the spectral descriptors — 'hann' (default, recommended to reduce leakage) or 'none' for a rectangular window.	'hann'
entropy_bins	int	Number of histogram bins for the entropy estimate. Defaults to 50.	50
fmin	float	Lowest frequency (Hz) considered for the dominant frequency and spectral centroid, excluding slow amplitude drift near DC. Defaults to 0.2.	0.2
fmax	float	Highest frequency (Hz) considered for those spectral descriptors. Defaults to 10.0.	10.0
save_data	bool	Save the descriptors to a data file. Defaults to True.	True
save_plot	bool	Save the figure (QoM time series + power spectrum). Defaults to True.	True
data_format	str	Data file format: 'csv', 'tsv' or 'txt'. Defaults to 'csv'.	'csv'
target_name	str	Output image name. Defaults to None (<name>_motiondescriptors.png).	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Type	Description
'MgFigure | None'	MgFigure | None: figure whose .data holds the scalar descriptors and the spectrum arrays
'MgFigure | None'	(frequencies, power), or None if there are too few frames.

Source code in musicalgestures/_motiondescriptors.py

def mg_motiondescriptors(self, window: str = 'hann', entropy_bins: int = 50,
                         fmin: float = 0.2, fmax: float = 10.0,
                         save_data: bool = True, save_plot: bool = True,
                         data_format: str = 'csv', target_name: str | None = None,
                         overwrite: bool = True) -> "MgFigure | None":
    """Scalar motion descriptors derived from the quantity-of-motion (QoM) signal.

    Computes a compact set of higher-level descriptors that summarise *how* something moves,
    complementing the per-frame motion data from :func:`motion`:

    - **motion_energy** — mean squared QoM; the overall amount of motion.
    - **motion_smoothness** — SPARC (spectral arc length) of the QoM profile; a dimensionless,
      validated smoothness metric (less negative = smoother, more negative = jerkier).
    - **motion_entropy** — normalised (0–1) Shannon entropy of the QoM magnitude distribution;
      the complexity/variedness of the motion.
    - **spectral descriptors** of the QoM signal (Hann-windowed by default): the **dominant
      frequency** (Hz, the dominant periodicity of the motion) and the **spectral centroid** (Hz, the
      "centre of mass" of the motion spectrum).

    Args:
        window (str, optional): FFT window for the spectral descriptors — 'hann' (default,
            recommended to reduce leakage) or 'none' for a rectangular window.
        entropy_bins (int, optional): Number of histogram bins for the entropy estimate. Defaults to 50.
        fmin (float, optional): Lowest frequency (Hz) considered for the dominant frequency and
            spectral centroid, excluding slow amplitude drift near DC. Defaults to 0.2.
        fmax (float, optional): Highest frequency (Hz) considered for those spectral descriptors.
            Defaults to 10.0.
        save_data (bool, optional): Save the descriptors to a data file. Defaults to True.
        save_plot (bool, optional): Save the figure (QoM time series + power spectrum). Defaults to True.
        data_format (str, optional): Data file format: 'csv', 'tsv' or 'txt'. Defaults to 'csv'.
        target_name (str, optional): Output image name. Defaults to None (``<name>_motiondescriptors.png``).
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgFigure | None: figure whose ``.data`` holds the scalar descriptors and the spectrum arrays
        (``frequencies``, ``power``), or None if there are too few frames.
    """
    qom, fps = _movement_qom(self)
    if qom.size < 4:
        print('Not enough frames to compute motion descriptors.')
        return None

    motion_energy = float(np.mean(qom ** 2))
    motion_smoothness = _sparc(qom, fps)
    motion_entropy = _motion_entropy(qom, bins=entropy_bins)
    freqs, power = _qom_spectrum(qom, fps, window=window)

    # Restrict the spectral descriptors to a motion band so slow amplitude drift near DC
    # doesn't masquerade as the dominant motion periodicity.
    band = (freqs >= fmin) & (freqs <= fmax)
    if band.any() and power[band].any():
        f_band, p_band = freqs[band], power[band]
        dominant_freq = float(f_band[int(np.argmax(p_band))])
        spectral_centroid = float(np.sum(f_band * p_band) / np.sum(p_band))
    else:
        dominant_freq = 0.0
        spectral_centroid = 0.0

    d = {
        'of': self.of,
        'fps': fps,
        'n_frames': int(qom.size),
        'motion_energy': motion_energy,
        'motion_smoothness': motion_smoothness,
        'motion_entropy': motion_entropy,
        'dominant_freq': dominant_freq,
        'spectral_centroid': spectral_centroid,
        'window': window,
        'fmin': fmin,
        'fmax': fmax,
        'qom': qom,
        'frequencies': freqs,
        'power': power,
    }

    if save_data:
        _save_descriptors(self.of, d, data_format, overwrite)

    target_name = resolve_filename(self.of, '_motiondescriptors.png', target_name, overwrite)

    times = np.arange(qom.size) / fps
    fig, (ax_q, ax_s) = plt.subplots(2, 1, figsize=(12, 7), dpi=300)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    ax_q.plot(times, qom, color='#1f77b4', lw=1.0)
    ax_q.fill_between(times, qom, color='#1f77b4', alpha=0.15)
    ax_q.set(xlabel='Time (s)', ylabel='Quantity of motion', title='Quantity of motion over time')
    ax_q.margins(x=0)

    # Power spectrum up to a sensible motion band (10 Hz), dominant frequency marked.
    band = freqs <= 10.0 if freqs.size else slice(None)
    ax_s.plot(freqs[band], power[band], color='#d62728', lw=1.0)
    ax_s.fill_between(freqs[band], power[band], color='#d62728', alpha=0.15)
    if dominant_freq > 0:
        ax_s.axvline(dominant_freq, color='#333333', ls='--', lw=1.0,
                     label=f'dominant {dominant_freq:.2f} Hz')
        ax_s.legend()
    ax_s.set(xlabel='Frequency (Hz)', ylabel='Power',
             title=f'QoM power spectrum ({window} window)')
    ax_s.margins(x=0)

    summary = (f'energy = {motion_energy:.3g}    smoothness (SPARC) = {motion_smoothness:.3f}    '
               f'entropy = {motion_entropy:.3f}    spectral centroid = {spectral_centroid:.2f} Hz')
    fig.suptitle(summary, fontsize=11, fontweight='bold')

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    if save_plot:
        plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    mgf = MgFigure(figure=fig, figure_type='video.motiondescriptors', data=d, layers=None,
                   image=target_name if save_plot else None)
    self.motiondescriptors_figure = mgf
    return mgf

phase_synchrony

phase_synchrony(fmin=0.5, fmax=4.0, fs=50.0, n_bins=36, dpi=300, autoshow=True, title=None, target_name=None, overwrite=True)

Quantify how phase-locked the motion is to the audio.

Both the audio onset-strength envelope and the quantity-of-motion envelope are band-pass filtered to the tempo band [fmin, fmax] Hz, and their instantaneous phases (via the Hilbert transform) are compared. The phase-locking value (PLV, 0–1) summarises the consistency of the audio↔motion phase difference; a polar histogram shows its distribution.

Returns an MgFigure (metrics in .data), or None if the video has no audio.

Source code in musicalgestures/_audio_video.py

def mg_phase_synchrony(self, fmin: float = 0.5, fmax: float = 4.0, fs: float = 50.0, n_bins: int = 36, dpi: int = 300,
                       autoshow: bool = True, title: str | None = None, target_name: str | None = None, overwrite: bool = True) -> "MgFigure | None":
    """
    Quantify how phase-locked the motion is to the audio.

    Both the audio onset-strength envelope and the quantity-of-motion envelope are
    band-pass filtered to the tempo band [``fmin``, ``fmax``] Hz, and their instantaneous phases
    (via the Hilbert transform) are compared. The **phase-locking value** (PLV, 0–1) summarises the
    consistency of the audio↔motion phase difference; a polar histogram shows its distribution.

    Returns an MgFigure (metrics in ``.data``), or None if the video has no audio.
    """
    from scipy.signal import butter, filtfilt, hilbert
    from musicalgestures._movementbeats import _movement_qom

    if not has_audio(self.filename):
        print('The video has no audio track.')
        return None

    target_name = resolve_filename(self.of, '_phase_synchrony.png', target_name, overwrite)

    oenv, t_a, sr = _audio_env(self, 'onset')
    qom, fps = _movement_qom(self)
    t_m = np.arange(len(qom)) / max(fps, 1e-9)
    t, a, m = _common_grid(t_a, oenv, t_m, qom, fs)
    if t is None or len(t) < 8:
        print('Audio and motion do not overlap enough in time.')
        return None

    ny = fs / 2.0
    b, aa = butter(2, [max(fmin, 0.01) / ny, min(fmax, ny - 0.01) / ny], btype='band')
    af = filtfilt(b, aa, _z(a))
    mf = filtfilt(b, aa, _z(m))
    pa = np.angle(hilbert(af))
    pm = np.angle(hilbert(mf))
    dphi = pm - pa
    z = np.exp(1j * dphi)
    plv = float(np.abs(np.mean(z)))
    mean_dphi = float(np.degrees(np.angle(np.mean(z))))

    fig = plt.figure(figsize=(12, 5), dpi=dpi)
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Audio–motion phase synchrony', fontsize=14)

    ax1 = fig.add_subplot(1, 2, 1)
    ax1.plot(t, af, color='#1f77b4', lw=0.8, label='Audio (band-passed)')
    ax1.plot(t, mf, color='#d62728', lw=0.8, alpha=0.8, label='Motion (band-passed)')
    ax1.set_xlabel('Time (s)')
    ax1.set_ylabel('Amplitude')
    ax1.legend(loc='upper right', fontsize=8)
    ax1.set_title(f'Tempo band {fmin}–{fmax} Hz', fontsize=10)

    ax2 = fig.add_subplot(1, 2, 2, projection='polar')
    counts, edges = np.histogram(dphi, bins=np.linspace(-np.pi, np.pi, n_bins + 1))
    centers = edges[:-1] + np.diff(edges) / 2
    cmax = counts.max() if counts.max() > 0 else 1
    ax2.bar(centers, counts, width=np.diff(edges), bottom=0.0,
            color=matplotlib.colormaps['viridis'](counts / cmax), alpha=0.85, edgecolor='none')
    ax2.plot([np.radians(mean_dphi), np.radians(mean_dphi)], [0, plv * cmax],
             color='crimson', lw=2, zorder=5)
    ax2.set_yticklabels([])
    ax2.set_title(f'Phase difference (motion − audio)\nPLV = {plv:.2f}, mean Δφ = {mean_dphi:.0f}°',
                  fontsize=10)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    d = {'plv': round(plv, 3), 'mean_phase_diff_deg': round(mean_dphi, 1),
         'fmin': fmin, 'fmax': fmax, 'fps': fps, 'sr': sr}
    mgf = MgFigure(figure=fig, figure_type='video.phase_synchrony', data=d, layers=None, image=target_name)
    self.phase_synchrony_figure = mgf
    return mgf

structure_comparison

structure_comparison(n=200, dpi=300, cmap='magma', autoshow=True, title=None, target_name=None, overwrite=True)

Compare the temporal structure of the audio with that of the video.

Builds a self-similarity matrix (SSM) of the audio (from MFCC frames) and of the video (from low-resolution frame appearance), resampled to the same n time points, and shows them side by side with their absolute difference map — bright regions in the difference are where the audio structure and the video structure diverge.

Returns an MgFigure (mean structural agreement in .data), or None if the video has no audio.

Source code in musicalgestures/_audio_video.py

def mg_structure_comparison(self, n: int = 200, dpi: int = 300, cmap: str = 'magma', autoshow: bool = True,
                            title: str | None = None, target_name: str | None = None, overwrite: bool = True) -> "MgFigure | None":
    """
    Compare the temporal **structure** of the audio with that of the video.

    Builds a self-similarity matrix (SSM) of the audio (from MFCC frames) and of the video
    (from low-resolution frame appearance), resampled to the same ``n`` time points, and shows
    them side by side with their absolute **difference map** — bright regions in the difference
    are where the audio structure and the video structure diverge.

    Returns an MgFigure (mean structural agreement in ``.data``), or None if the video has no audio.
    """
    import librosa

    if not has_audio(self.filename):
        print('The video has no audio track.')
        return None

    target_name = resolve_filename(self.of, '_structure_comparison.png', target_name, overwrite)

    # Audio feature matrix (MFCC) → resample to n columns
    y, sr = self._load()
    mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=20, hop_length=self.hop_length)  # (20, Ta)
    idx_a = np.linspace(0, mfcc.shape[1] - 1, n).astype(int)
    audio_feat = mfcc[:, idx_a].T  # (n, 20)

    # Motion feature: low-res grayscale frame appearance at n evenly spaced frames
    cap = cv2.VideoCapture(self.filename)
    total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) or 1
    want = set(np.linspace(0, total - 1, min(n, total)).astype(int).tolist())
    feats, i = [], 0
    while True:
        ret, frame = cap.read()
        if not ret:
            break
        if i in want:
            small = cv2.resize(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY), (32, 32)).astype(np.float32)
            feats.append(small.ravel())
        i += 1
    cap.release()
    motion_feat = np.array(feats) if feats else np.zeros((n, 1024))
    # Match lengths (resample motion rows to n)
    if motion_feat.shape[0] != n and motion_feat.shape[0] > 1:
        ridx = np.linspace(0, motion_feat.shape[0] - 1, n).astype(int)
        motion_feat = motion_feat[ridx]

    ssm_audio = _ssm_from_features(audio_feat)
    ssm_motion = _ssm_from_features(motion_feat)
    k = min(ssm_audio.shape[0], ssm_motion.shape[0])
    ssm_audio, ssm_motion = ssm_audio[:k, :k], ssm_motion[:k, :k]
    diff = np.abs(ssm_audio - ssm_motion)
    agreement = float(1.0 - diff.mean())

    fig, axes = plt.subplots(1, 3, figsize=(15, 5), dpi=dpi)
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Audio vs video structural similarity', fontsize=14)
    for ax, mat, ttl, cm in zip(axes, [ssm_audio, ssm_motion, diff],
                                ['Audio SSM (MFCC)', 'Video SSM (appearance)',
                                 f'|difference|  (agreement {agreement:.2f})'],
                                [cmap, cmap, 'inferno']):
        im = ax.imshow(mat, origin='lower', cmap=cm, aspect='equal')
        ax.set_title(ttl, fontsize=10)
        ax.set_xticks([]); ax.set_yticks([])
        fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    d = {'structural_agreement': round(agreement, 3), 'n': int(k), 'sr': sr}
    mgf = MgFigure(figure=fig, figure_type='video.structure_comparison', data=d, layers=None, image=target_name)
    self.structure_comparison_figure = mgf
    return mgf

motion_audio_coupling

motion_audio_coupling(dpi=300, cmap='coolwarm', dot_size=260, autoshow=True, title=None, target_name=None, overwrite=True, **pose_kwargs)

Map which body parts' speeds correlate most strongly with the music's onsets.

For every pose marker the per-frame speed is correlated with the audio onset-strength envelope (sampled at the video frame rate). The result is shown as a body map — the average pose with each marker coloured by its correlation — plus a sorted bar chart, and a CSV of the per-marker correlations. Uses cached pose keypoints when available, otherwise runs pose() first (**pose_kwargs are forwarded).

Returns an MgFigure (per-marker correlations in .data), or None if the video has no audio.

Source code in musicalgestures/_audio_video.py

def mg_motion_audio_coupling(self, dpi: int = 300, cmap: str = 'coolwarm', dot_size: int = 260, autoshow: bool = True,
                           title: str | None = None, target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure | None":
    """
    Map which body parts' speeds correlate most strongly with the music's onsets.

    For every pose marker the per-frame speed is correlated with the audio onset-strength
    envelope (sampled at the video frame rate). The result is shown as a body map — the average
    pose with each marker coloured by its correlation — plus a sorted bar chart, and a CSV of the
    per-marker correlations. Uses cached pose keypoints when available, otherwise runs ``pose()``
    first (``**pose_kwargs`` are forwarded).

    Returns an MgFigure (per-marker correlations in ``.data``), or None if the video has no audio.
    """
    from musicalgestures._pose_visualize import _positions_from_data

    if not has_audio(self.filename):
        print('The video has no audio track.')
        return None

    if getattr(self, '_pose_keypoints', None) is None:
        pose_kwargs.setdefault('save_video', False)
        pose_kwargs.setdefault('save_average_pose', False)
        pose_kwargs.setdefault('save_trajectories', False)
        self.pose(**pose_kwargs)
    c = self._pose_keypoints
    names, connections = c['names'], c.get('connections') or []
    width, height, fps = c['width'], c['height'], c['fps']

    target_name = resolve_filename(c['of'], '_motion_audio_coupling.png', target_name, overwrite)

    coords, _ = _positions_from_data(c['data'], len(names))  # (T, n, 2) normalised
    px = coords * np.array([width, height])
    speed = np.sqrt((np.diff(px, axis=0) ** 2).sum(axis=2))   # (T-1, n) px/frame

    # Audio onset envelope sampled at the video frame times (length T), then aligned to speed.
    oenv, t_a, sr = _audio_env(self, 'onset')
    t_frames = np.arange(coords.shape[0]) / max(fps, 1e-9)
    aud_per_frame = np.interp(t_frames, t_a, oenv)[1:]  # align to diff length

    corrs = np.array([_safe_corr(np.nan_to_num(speed[:, i]), aud_per_frame) for i in range(len(names))])

    mean_px = np.nanmean(px, axis=0)
    vmax = float(np.nanmax(np.abs(corrs))) if np.isfinite(corrs).any() else 1.0
    vmax = max(vmax, 1e-6)
    cmap_obj = matplotlib.colormaps[cmap]
    norm = matplotlib.colors.Normalize(vmin=-vmax, vmax=vmax)

    fig, (axb, axbar) = plt.subplots(1, 2, figsize=(14, 7), dpi=dpi,
                                     gridspec_kw={'width_ratios': [1, 1]})
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Per-body-part coupling to the music', fontsize=14)

    # Body map
    for a, b in connections:
        if a < len(names) and b < len(names) and not (np.isnan(mean_px[a]).any() or np.isnan(mean_px[b]).any()):
            axb.plot([mean_px[a, 0], mean_px[b, 0]], [mean_px[a, 1], mean_px[b, 1]],
                     color='#bbbbbb', lw=2, zorder=1)
    for i in range(len(names)):
        if not np.isnan(mean_px[i]).any():
            axb.scatter(mean_px[i, 0], mean_px[i, 1], s=dot_size, c=[cmap_obj(norm(corrs[i]))],
                        edgecolors='black', linewidths=0.6, zorder=2)
    axb.set_xlim(0, width); axb.set_ylim(height, 0); axb.set_aspect('equal'); axb.axis('off')
    axb.set_title('Body map (marker colour = correlation of speed with audio)', fontsize=10)
    fig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap_obj), ax=axb,
                 fraction=0.046, pad=0.04, label='Pearson r')

    # Sorted bar chart
    order = np.argsort(corrs)
    axbar.barh(np.arange(len(names)), corrs[order],
               color=[cmap_obj(norm(corrs[o])) for o in order])
    axbar.set_yticks(np.arange(len(names)))
    axbar.set_yticklabels([names[o] for o in order], fontsize=5)
    axbar.axvline(0, color='#888888', lw=0.8)
    axbar.set_xlabel('Correlation of marker speed with audio onset (Pearson r)')
    axbar.set_title('Per-marker coupling, ranked', fontsize=10)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    stats = [{'Marker': names[i], 'Correlation': round(float(corrs[i]), 3)} for i in range(len(names))]
    try:
        import pandas as pd
        pd.DataFrame(stats).to_csv(os.path.splitext(target_name)[0] + '.csv', index=False)
    except Exception as e:
        print(f'Warning: could not save CSV: {e}')

    d = {'correlations': stats, 'mean_abs_correlation': round(float(np.nanmean(np.abs(corrs))), 3),
         'fps': fps, 'sr': sr}
    mgf = MgFigure(figure=fig, figure_type='video.motion_audio_coupling', data=d, layers=None, image=target_name)
    self.motion_audio_coupling_figure = mgf
    return mgf

dynamics_coupling

dynamics_coupling(fs=50.0, max_lag=2.0, dpi=300, autoshow=True, title=None, target_name=None, overwrite=True)

Compare the audio RMS level with the quantity of motion — does the dancer move more when the music is louder?

Aligns the audio RMS envelope with the quantity-of-motion envelope and reports their correlation (at zero lag and at the best lag within max_lag seconds). The figure overlays the two normalised envelopes and shows a scatter of RMS level vs. motion.

Returns an MgFigure (metrics in .data), or None if the video has no audio.

Source code in musicalgestures/_audio_video.py

def mg_dynamics_coupling(self, fs: float = 50.0, max_lag: float = 2.0, dpi: int = 300, autoshow: bool = True,
                         title: str | None = None, target_name: str | None = None, overwrite: bool = True) -> "MgFigure | None":
    """
    Compare the audio **RMS level** with the **quantity of motion** — does the dancer move more
    when the music is louder?

    Aligns the audio RMS envelope with the quantity-of-motion envelope and reports their
    correlation (at zero lag and at the best lag within ``max_lag`` seconds). The figure overlays
    the two normalised envelopes and shows a scatter of RMS level vs. motion.

    Returns an MgFigure (metrics in ``.data``), or None if the video has no audio.
    """
    from musicalgestures._movementbeats import _movement_qom

    if not has_audio(self.filename):
        print('The video has no audio track.')
        return None

    target_name = resolve_filename(self.of, '_dynamics_coupling.png', target_name, overwrite)

    rms, t_a, sr = _audio_env(self, 'rms')
    qom, fps = _movement_qom(self)
    t_m = np.arange(len(qom)) / max(fps, 1e-9)
    t, a, m = _common_grid(t_a, rms, t_m, qom, fs)
    if t is None or len(t) < 8:
        print('Audio and motion do not overlap enough in time.')
        return None
    az, mz = _z(a), _z(m)

    zero_r = _safe_corr(az, mz)
    # Best lag within +/- max_lag seconds
    max_shift = int(max_lag * fs)
    best_r, best_lag = zero_r, 0.0
    for s in range(-max_shift, max_shift + 1):
        if s < 0:
            r = _safe_corr(az[-s:], mz[:len(mz) + s])
        elif s > 0:
            r = _safe_corr(az[:len(az) - s], mz[s:])
        else:
            r = zero_r
        if r > best_r:
            best_r, best_lag = r, s / fs

    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5), dpi=dpi,
                                   gridspec_kw={'width_ratios': [2, 1]})
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Audio RMS level vs. quantity of motion', fontsize=14)

    ax1.plot(t, az, color='#1f77b4', lw=0.9, label='Audio RMS level')
    ax1.plot(t, mz, color='#d62728', lw=0.9, alpha=0.8, label='Quantity of motion')
    ax1.set_xlabel('Time (s)'); ax1.set_ylabel('Normalised')
    ax1.legend(loc='upper right', fontsize=8)
    ax1.set_title(f'Zero-lag r = {zero_r:.2f}   |   best r = {best_r:.2f} @ {best_lag:+.2f}s',
                  fontsize=10)

    ax2.scatter(az, mz, s=4, alpha=0.3, color='#444444')
    ax2.set_xlabel('Audio RMS (z)'); ax2.set_ylabel('Quantity of motion (z)')
    ax2.set_title('RMS level vs. motion', fontsize=10)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    d = {'zero_lag_corr': round(zero_r, 3), 'best_corr': round(best_r, 3),
         'best_lag_s': round(best_lag, 3), 'fps': fps, 'sr': sr}
    mgf = MgFigure(figure=fig, figure_type='video.dynamics_coupling', data=d, layers=None, image=target_name)
    self.dynamics_coupling_figure = mgf
    return mgf

pose

pose(model='mediapipe', device='gpu', threshold=0.1, downsampling_factor=2, use_cache=True, save_data=True, data_format='csv', save_video=True, style='both', overlay=True, background='black', convert=None, quiet=True, marker_history=0, save_average_pose=True, save_trajectories=True, transparent_trajectories=None, trajectory_background=None, trajectory_labels=False, target_name_video=None, target_name_data=None, target_name_average=None, target_name_trajectories=None, overwrite=True)

Renders a video with the pose estimation (aka. "keypoint detection" or "skeleton tracking") overlaid on it. Outputs the predictions in a text file containing the normalized x and y coordinates of each keypoint (default format is csv).

Supports two backends:

• MediaPipe (model='mediapipe'): Uses Google's MediaPipe Pose which detects 33 landmarks. Runs on CPU, or on GPU via MediaPipe's GPU delegate when device='gpu' (with automatic CPU fallback if the delegate is unavailable). Requires the optional mediapipe package (pip install musicalgestures[pose]). On first use, the model file (~8–28 MB) is downloaded automatically and cached in musicalgestures/models/.
• OpenPose (model='body_25', 'coco', or 'mpi'): Uses Caffe-based OpenPose models. Model weights (~200 MB) are downloaded on first use. GPU here requires an OpenCV built with CUDA; if unavailable while device='gpu', pose() automatically switches to the MediaPipe backend (when installed) for GPU acceleration.

Parameters:

Name	Type	Description	Default
model	str	Pose model to use. 'mediapipe' (default) uses MediaPipe Pose (33 landmarks with depth + visibility, model auto-downloaded on first use); it is fast on plain CPU, needs no CUDA build, and is best for single-person analysis. 'body_25' loads the OpenPose BODY_25 model (25 keypoints), 'mpi' loads the MPII model (15 keypoints), 'coco' loads the COCO model (18 keypoints). The OpenPose models support multi-person scenes but are slow without a CUDA-enabled OpenCV build. Defaults to 'mediapipe'.	'mediapipe'
device	str	Compute backend ('cpu' or 'gpu'). For OpenPose models this selects the OpenCV DNN backend (GPU needs a CUDA-enabled OpenCV). For MediaPipe it selects the inference delegate (GPU delegate with CPU fallback). Defaults to 'gpu'.	'gpu'
threshold	float	The normalized confidence threshold that decides whether we keep or discard a predicted point. Discarded points get substituted with (0, 0) in the output data. Defaults to 0.1.	0.1
downsampling_factor	int	Decides how much we downsample the video before we pass it to the neural network. Ignored when model='mediapipe'. Defaults to 2.	2
use_cache	bool	If True (default), reuse keypoints from a previous pose() run on this object (same model/threshold) to re-render a different style/overlay/background without re-running the network — e.g. run style='markers' then style='skeleton' fast. Defaults to True.	True
save_data	bool	Whether we save the predicted pose data to a file. Defaults to True.	True
data_format	str	Specifies format of pose-data. Accepted values are 'csv', 'tsv', 'txt' and 'c3d' (motion-capture format; requires the optional c3d package). For multiple output formats, use a list, e.g. ['csv', 'c3d']. Defaults to 'csv'.	'csv'
save_video	bool	Whether we save the video with the estimated pose overlaid on it. Defaults to True.	True
style	str	How to draw the pose. 'both' draws markers (keypoints) connected by joint lines (the skeleton); 'markers' draws only the keypoints; 'skeleton' draws only the connecting joint lines. Defaults to 'both'.	'both'
overlay	bool	If True, draw the pose on top of the original video frames. If False, draw it on a plain background instead (a "markers only" video with no video underneath). Defaults to True.	True
background	str	Background colour used when overlay=False: 'black' (default) or 'white'. With 'white' the skeleton and markers are drawn in black (an inverted, print-friendly look). With 'black' they are drawn in bright colours. Ignored when overlay=True.	'black'
marker_history	int	If greater than 0, draw a motion trail for every marker by joining its positions over the last marker_history frames. Defaults to 0 (no trails). Works in all rendering paths (OpenPose, MediaPipe, and cached re-render).	0
convert	bool	Whether non-AVI input is first converted to an all-intra MJPEG .avi (cached as self.as_avi) for frame-accurate decoding. Defaults to None ("auto"): the MediaPipe backend reads the source file directly (it decodes sequentially through an FFmpeg pipe and needs no intra-frame AVI), while the OpenPose backend converts. Pass True/False to force the behaviour.	None
quiet	bool	MediaPipe only. If True (default), suppress MediaPipe's native C++/GL console logs (EGL init, absl INFO/WARNING, GPU-delegate messages) during inference. Set to False to see them for debugging.	True
target_name_video	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_pose" should be used).	None
save_average_pose	bool	Whether to also render an image of the average pose over the whole video, with each marker coloured/labelled by its average quantity of motion (px/frame) and labelled with its dominant motion frequency (Hz). A CSV of the per-marker statistics is saved alongside it. Defaults to True.	True
save_trajectories	bool	Whether to also render an image of every marker's spatial trajectory across the whole video. Defaults to True.	True
trajectory_labels	bool	Whether to annotate the trajectories image with each marker's name. Defaults to False (cleaner image).	False
trajectory_background	str	Background of the trajectories PNG: 'black', 'white', or 'transparent' (for overlaying on the video). Defaults to None ("auto"): transparent when the trajectories image is the only one exported, else black. Takes precedence over the legacy transparent_trajectories flag.	None
target_name_data	str	Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_pose" should be used).	None
target_name_average	str	Target output name for the average-pose image. Defaults to None (input filename with the suffix "_pose_average.png").	None
target_name_trajectories	str	Target output name for the trajectories image. Defaults to None (input filename with the suffix "_pose_trajectories.png").	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	'musicalgestures.MgVideo'	An MgVideo pointing to the output video. The average-pose and trajectories images (when rendered) are attached as .average_pose and .trajectories (MgImage), and the collected keypoints are available on the parent object as self.pose_average_image / self.pose_trajectories_image.

Source code in musicalgestures/_pose.py

def pose(
        self: "musicalgestures.MgVideo",
        model: str = 'mediapipe',
        device: str = 'gpu',
        threshold: float = 0.1,
        downsampling_factor: int = 2,
        use_cache: bool = True,
        save_data: bool = True,
        data_format: str | list = 'csv',
        save_video: bool = True,
        style: str = 'both',
        overlay: bool = True,
        background: str = 'black',
        convert: bool | None = None,
        quiet: bool = True,
        marker_history: int = 0,
        save_average_pose: bool = True,
        save_trajectories: bool = True,
        transparent_trajectories: bool | None = None,
        trajectory_background: str | None = None,
        trajectory_labels: bool = False,
        target_name_video: str | None = None,
        target_name_data: str | None = None,
        target_name_average: str | None = None,
        target_name_trajectories: str | None = None,
        overwrite: bool = True) -> "musicalgestures.MgVideo":
    """
    Renders a video with the pose estimation (aka. "keypoint detection" or "skeleton tracking") overlaid on it.
    Outputs the predictions in a text file containing the normalized x and y coordinates of each keypoint
    (default format is csv).

    Supports two backends:

    * **MediaPipe** (``model='mediapipe'``): Uses Google's MediaPipe Pose which detects 33
      landmarks. Runs on CPU, or on GPU via MediaPipe's GPU delegate when ``device='gpu'``
      (with automatic CPU fallback if the delegate is unavailable). Requires the optional
      ``mediapipe`` package (``pip install musicalgestures[pose]``). On first use, the model
      file (~8–28 MB) is downloaded automatically and cached in ``musicalgestures/models/``.
    * **OpenPose** (``model='body_25'``, ``'coco'``, or ``'mpi'``): Uses Caffe-based OpenPose
      models.  Model weights (~200 MB) are downloaded on first use. GPU here requires an
      OpenCV built with CUDA; if unavailable while ``device='gpu'``, ``pose()`` automatically
      switches to the MediaPipe backend (when installed) for GPU acceleration.

    Args:
        model (str, optional): Pose model to use. ``'mediapipe'`` (default) uses MediaPipe Pose (33
            landmarks with depth + visibility, model auto-downloaded on first use); it is fast on plain
            CPU, needs no CUDA build, and is best for single-person analysis. ``'body_25'`` loads the
            OpenPose BODY_25 model (25 keypoints), ``'mpi'`` loads the MPII model (15 keypoints),
            ``'coco'`` loads the COCO model (18 keypoints). The OpenPose models support multi-person
            scenes but are slow without a CUDA-enabled OpenCV build. Defaults to 'mediapipe'.
        device (str, optional): Compute backend ('cpu' or 'gpu'). For OpenPose models this
            selects the OpenCV DNN backend (GPU needs a CUDA-enabled OpenCV). For MediaPipe
            it selects the inference delegate (GPU delegate with CPU fallback). Defaults to 'gpu'.
        threshold (float, optional): The normalized confidence threshold that decides whether we
            keep or discard a predicted point. Discarded points get substituted with (0, 0) in the
            output data. Defaults to 0.1.
        downsampling_factor (int, optional): Decides how much we downsample the video before we
            pass it to the neural network. Ignored when ``model='mediapipe'``. Defaults to 2.
        use_cache (bool, optional): If True (default), reuse keypoints from a previous pose() run on
            this object (same model/threshold) to re-render a different `style`/`overlay`/`background`
            without re-running the network — e.g. run `style='markers'` then `style='skeleton'` fast.
            Defaults to True.
        save_data (bool, optional): Whether we save the predicted pose data to a file. Defaults to True.
        data_format (str, optional): Specifies format of pose-data. Accepted values are 'csv', 'tsv',
            'txt' and 'c3d' (motion-capture format; requires the optional ``c3d`` package). For multiple
            output formats, use a list, e.g. ['csv', 'c3d']. Defaults to 'csv'.
        save_video (bool, optional): Whether we save the video with the estimated pose overlaid on it.
            Defaults to True.
        style (str, optional): How to draw the pose. `'both'` draws markers (keypoints) connected by
            joint lines (the skeleton); `'markers'` draws only the keypoints; `'skeleton'` draws only
            the connecting joint lines. Defaults to 'both'.
        overlay (bool, optional): If True, draw the pose on top of the original video frames. If False,
            draw it on a plain background instead (a "markers only" video with no video underneath).
            Defaults to True.
        background (str, optional): Background colour used when `overlay=False`: `'black'` (default) or
            `'white'`. With `'white'` the skeleton and markers are drawn in black (an inverted, print-friendly
            look). With `'black'` they are drawn in bright colours. Ignored when `overlay=True`.
        marker_history (int, optional): If greater than 0, draw a motion trail for every marker by joining its
            positions over the last `marker_history` frames. Defaults to 0 (no trails). Works in all rendering
            paths (OpenPose, MediaPipe, and cached re-render).
        convert (bool, optional): Whether non-AVI input is first converted to an all-intra MJPEG `.avi`
            (cached as ``self.as_avi``) for frame-accurate decoding. Defaults to None ("auto"): the
            MediaPipe backend reads the source file directly (it decodes sequentially through an FFmpeg
            pipe and needs no intra-frame AVI), while the OpenPose backend converts. Pass True/False to
            force the behaviour.
        quiet (bool, optional): MediaPipe only. If True (default), suppress MediaPipe's native C++/GL
            console logs (EGL init, absl INFO/WARNING, GPU-delegate messages) during inference. Set to
            False to see them for debugging.
        target_name_video (str, optional): Target output name for the video. Defaults to None (which
            assumes that the input filename with the suffix "_pose" should be used).
        save_average_pose (bool, optional): Whether to also render an image of the average pose over
            the whole video, with each marker coloured/labelled by its average quantity of motion
            (px/frame) and labelled with its dominant motion frequency (Hz). A CSV of the per-marker
            statistics is saved alongside it. Defaults to True.
        save_trajectories (bool, optional): Whether to also render an image of every marker's spatial
            trajectory across the whole video. Defaults to True.
        trajectory_labels (bool, optional): Whether to annotate the trajectories image with each
            marker's name. Defaults to False (cleaner image).
        trajectory_background (str, optional): Background of the trajectories PNG: ``'black'``,
            ``'white'``, or ``'transparent'`` (for overlaying on the video). Defaults to None
            ("auto"): transparent when the trajectories image is the only one exported, else black.
            Takes precedence over the legacy ``transparent_trajectories`` flag.
        target_name_data (str, optional): Target output name for the data. Defaults to None (which
            assumes that the input filename with the suffix "_pose" should be used).
        target_name_average (str, optional): Target output name for the average-pose image. Defaults
            to None (input filename with the suffix "_pose_average.png").
        target_name_trajectories (str, optional): Target output name for the trajectories image.
            Defaults to None (input filename with the suffix "_pose_trajectories.png").
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically
            increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: An MgVideo pointing to the output video. The average-pose and trajectories images
            (when rendered) are attached as ``.average_pose`` and ``.trajectories`` (MgImage), and the
            collected keypoints are available on the parent object as
            ``self.pose_average_image`` / ``self.pose_trajectories_image``.
    """

    style = str(style).lower()
    if style not in ('both', 'markers', 'skeleton'):
        print(f"Unrecognized style '{style}', falling back to 'both'. Use 'both', 'markers' or 'skeleton'.")
        style = 'both'

    background = str(background).lower()
    if background not in ('black', 'white'):
        print(f"Unrecognized background '{background}', falling back to 'black'. Use 'black' or 'white'.")
        background = 'black'

    # --- MediaPipe backend ---------------------------------------------------
    # Explicit MediaPipe request, or auto-preference: when GPU is requested for an
    # OpenPose model but OpenCV has no CUDA backend, fall back to the (fast, reliable)
    # MediaPipe pose backend instead of CPU OpenPose. We use the CPU delegate here:
    # MediaPipe's GPU delegate is the OpenGL-ES path (the integrated GPU on Linux, not
    # an NVIDIA card) and is fragile, so it is reserved for explicit model='mediapipe',
    # device='gpu' requests.
    use_mediapipe = model.lower() == 'mediapipe'
    mediapipe_device = device
    if not use_mediapipe and device.lower() == 'gpu' and not in_colab() and get_cuda_device_count() <= 0:
        if _mediapipe_available():
            print(
                f"GPU requested but OpenCV has no CUDA backend; switching from '{model}' to the "
                "MediaPipe pose backend (CPU/XNNPACK — fast and reliable).\n  "
                + cuda_unavailable_reason()
            )
            use_mediapipe = True
            mediapipe_device = 'cpu'
        # else: fall through to the OpenPose path, which will warn and use CPU.

    # MediaPipe is an optional dependency (the `[pose]` extra). If it was selected (it is the
    # default backend) but isn't installed, fall back to the OpenPose BODY_25 model — it runs on
    # the always-present OpenCV DNN and auto-downloads its weights — so pose() works on a bare
    # `pip install musicalgestures`.
    #
    # That fallback assumed OpenCV could always load a Caffe model, which
    # stopped being true in OpenCV 5.0. Where it is not true there is nowhere
    # to fall back *to*, and announcing a fallback that then fails on the
    # backend it fell back to is worse than saying so plainly.
    if use_mediapipe and not _mediapipe_available():
        if not caffe_supported():
            raise MgDependencyError(
                f"pose() needs MediaPipe on this machine. The default backend "
                f"is not installed, and the OpenPose fallback cannot run "
                f"either: its models are Caffe, and OpenCV {cv2.__version__} "
                f"dropped the Caffe importer in 5.0. Install it with "
                f"`pip install musicalgestures[pose]`, or install OpenCV 4 "
                f"(`pip install 'opencv-python<5'`) to use the OpenPose "
                f"skeletons instead."
            )
        print("MediaPipe is not installed; falling back to the OpenPose 'body_25' backend. "
              "Install MediaPipe for the default backend with: pip install musicalgestures[pose]")
        use_mediapipe = False
        if model.lower() == 'mediapipe':
            model = 'body_25'

    # Resolve the "auto" convert default: MediaPipe reads the source directly through an
    # FFmpeg pipe and needs no all-intra AVI; OpenPose keeps the frame-accurate conversion.
    if convert is None:
        convert = not use_mediapipe

    # --- Reuse cached keypoints (skip re-inference) --------------------------
    # If a previous pose() run on this object used the same model/threshold, re-render
    # from the stored keypoints with the new style/overlay/background instead of running
    # the (expensive) network again.
    effective_model = 'mediapipe' if use_mediapipe else model.lower()
    cache = getattr(self, '_pose_keypoints', None)
    if use_cache and cache is not None \
            and cache.get('model') == effective_model \
            and cache.get('threshold') == threshold \
            and cache.get('downsampling_factor') == (None if use_mediapipe else downsampling_factor):
        return _rerender_pose_from_cache(
            self, style=style, overlay=overlay, background=background,
            save_data=save_data, data_format=data_format, save_video=save_video,
            save_average_pose=save_average_pose, save_trajectories=save_trajectories,
            transparent_trajectories=transparent_trajectories,
            trajectory_background=trajectory_background, trajectory_labels=trajectory_labels,
            marker_history=marker_history,
            target_name_video=target_name_video, target_name_data=target_name_data,
            target_name_average=target_name_average, target_name_trajectories=target_name_trajectories,
            overwrite=overwrite)

    if use_mediapipe:
        return _pose_mediapipe(
            self,
            device=mediapipe_device,
            threshold=threshold,
            save_data=save_data,
            data_format=data_format,
            save_video=save_video,
            style=style,
            overlay=overlay,
            background=background,
            convert=convert,
            quiet=quiet,
            marker_history=marker_history,
            save_average_pose=save_average_pose,
            save_trajectories=save_trajectories,
            transparent_trajectories=transparent_trajectories,
            trajectory_background=trajectory_background,
            trajectory_labels=trajectory_labels,
            target_name_video=target_name_video,
            target_name_data=target_name_data,
            target_name_average=target_name_average,
            target_name_trajectories=target_name_trajectories,
            overwrite=overwrite,
        )
    # -------------------------------------------------------------------------

    # The OpenPose backends are Caffe models, and OpenCV removed its Caffe
    # importer in 5.0 -- `readNetFromCaffe` is gone and `readNet` refuses the
    # format outright. Checked here, before the weights are looked for, so an
    # incompatible environment is not discovered after a 200 MB download and
    # a full decode; the failure used to surface as `AttributeError: module
    # 'cv2.dnn' has no attribute 'readNetFromCaffe'` from deep inside the run.
    #
    # Not silently switched to MediaPipe. Its 33 landmarks are a different
    # skeleton from BODY_25's, COCO's or MPI's, so a substituted backend would
    # return data that looks like what was asked for and is not.
    _require_caffe_support()

    module_path = os.path.abspath(os.path.dirname(musicalgestures.__file__))

    if model.lower() == 'mpi':
        protoFile = module_path + '/pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt'
        weightsFile = module_path + '/pose/mpi/pose_iter_160000.caffemodel'
        model = 'mpi'
        nPoints = 15
        POSE_PAIRS = [[0, 1], [1, 2], [2, 3], [3, 4], [1, 5], [5, 6], [6, 7], [
            1, 14], [14, 8], [8, 9], [9, 10], [14, 11], [11, 12], [12, 13]]
    elif model.lower() == 'coco':
        protoFile = module_path + '/pose/coco/pose_deploy_linevec.prototxt'
        weightsFile = module_path + '/pose/coco/pose_iter_440000.caffemodel'
        model = 'coco'
        nPoints = 18
        POSE_PAIRS = [[1, 0], [1, 2], [1, 5], [2, 3], [3, 4], [5, 6], [6, 7], [1, 8], [
            8, 9], [9, 10], [1, 11], [11, 12], [12, 13], [0, 14], [0, 15], [14, 16], [15, 17]]
    elif model.lower() == 'body_25':
        protoFile = module_path + '/pose/body_25/pose_deploy.prototxt'
        weightsFile = module_path + '/pose/body_25/pose_iter_584000.caffemodel'
        model = 'body_25'
        nPoints = 25
        POSE_PAIRS = [[1, 8], [1, 2], [1, 5], [2, 3], [3, 4], [5, 6], [6, 7], [8, 9], [9, 10], [10, 11], [8, 12], [12, 13], [
            13, 14], [1, 0], [0, 15], [15, 17], [0, 16], [16, 18], [14, 19], [19, 20], [14, 21], [11, 22], [22, 23], [11, 24]]
    else:
        print(f'Unrecognized model "{model}", switching to default (mpi).')
        protoFile = module_path + '/pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt'
        weightsFile = module_path + '/pose/mpi/pose_iter_160000.caffemodel'
        model = 'mpi'

    # Check if .caffemodel file exists, download if necessary
    if not os.path.exists(weightsFile):
        print('Could not find weights file.')
        # Notebook/nbclient runs cannot satisfy input(), so auto-download in non-interactive mode.
        if not sys.stdin or not sys.stdin.isatty():
            print('Non-interactive session detected. Downloading model weights automatically (~200MB).')
            download_model(model)
        else:
            print('Do you want to download it (~200MB)? (y/n)')
            answer = input()
            if answer.lower() == 'n':
                print('Ok. Exiting...')
                return musicalgestures.MgVideo(self.filename, color=self.color, returned_by_process=True)
            elif answer.lower() == 'y':
                download_model(model)
            else:
                print(f'Unrecognized answer "{answer}". Exiting...')
                return musicalgestures.MgVideo(self.filename, color=self.color, returned_by_process=True)

        if not os.path.exists(weightsFile):
            #: A print-and-return here left the caller's next line failing with
            #: a bare AttributeError on a result attribute that was never set
            #: (reported from a Colab run on 2026-09-03). One clear error that
            #: names the fix beats two confusing ones.
            raise RuntimeError(
                f"the {model} model weights could not be downloaded, so pose() cannot "
                "run. The MediaPipe backend needs no weight download and is the "
                "default when installed: pip install \"musicalgestures[pose]\" "
                "(on Colab, run that in a cell and restart the runtime if needed).")

    # Read the network into Memory
    net = cv2.dnn.readNetFromCaffe(protoFile, weightsFile)
    device = device.lower()
    # enforce CPU device in Colab
    if in_colab() and device == 'gpu':
        print('Sorry, OpenCV GPU acceleration is not supported in Colab. Switching to CPU.')
        device = 'cpu'
    elif device == 'gpu':
        if get_cuda_device_count() <= 0:
            print('OpenCV CUDA backend is unavailable. Switching to CPU.\n  ' + cuda_unavailable_reason())
            device = 'cpu'

    if device == "cpu":
        net.setPreferableBackend(cv2.dnn.DNN_TARGET_CPU)
    elif device == "gpu":
        net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
        net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)
    else:
        print(f'Unrecognized device "{device}", switching to default (cpu).')
        net.setPreferableBackend(cv2.dnn.DNN_TARGET_CPU)

    of, fex = os.path.splitext(self.filename)
    # Write the result in the original container so we don't produce an .avi that then has
    # to be converted to .mp4; the AVI is only an intermediate for frame-accurate decoding.
    output_fex = fex

    if convert and fex.lower() != '.avi':
        # first check if there already is a converted version, if not create one and register it to the parent self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex
        filename = of + fex
    else:
        # use the source file directly (e.g. an mp4 that decodes frame-accurately)
        filename = self.filename

    inWidth = int(roundup(self.width/downsampling_factor, 2))
    inHeight = int(roundup(self.height/downsampling_factor, 2))

    pb = MgProgressbar(total=self.length, prefix='Rendering pose estimation video:')

    if save_video:
        if target_name_video is None:
            target_name_video = of + '_pose' + output_fex
        else:
            target_name_video = os.path.splitext(target_name_video)[0] + output_fex
        if not overwrite:
            target_name_video = generate_outfilename(target_name_video)

    # Pipe video with FFmpeg for reading frame by frame
    cmd = ['ffmpeg', '-y', '-i', filename] # define ffmpeg command        
    process = ffmpeg_cmd(cmd, total_time=self.length, pipe='read')
    video_out = None

    ii = 0
    data = []
    # Accumulate the average frame as a background for the average-pose image
    collect_extra = save_average_pose or save_trajectories
    avg_acc = np.zeros((self.height, self.width, 3), dtype=np.float64) if save_average_pose else None
    avg_n = 0
    from collections import deque
    _trail: deque | None = deque(maxlen=int(marker_history)) if marker_history and marker_history > 0 else None

    while True:
        # Read frame-by-frame
        out = process.stdout.read(self.width*self.height*3)

        if out == b'':
            pb.progress(self.length)
            break

        # Transform the bytes read into a numpy array
        frame = np.frombuffer(out, dtype=np.uint8).reshape([self.height, self.width, 3]).copy() # height, width, channels

        if avg_acc is not None:
            avg_acc += frame
            avg_n += 1

        inpBlob = cv2.dnn.blobFromImage(frame, 1.0 / 255, (inWidth, inHeight), (0, 0, 0), swapRB=False, crop=False)
        net.setInput(inpBlob)
        output = net.forward()

        H = output.shape[2]
        W = output.shape[3]
        points: list[tuple[int, int] | None] = []

        for i in range(nPoints):

            # confidence map of corresponding body's part.
            probMap = output[0, i, :, :]

            # Find global maxima of the probMap.
            minVal, prob, minLoc, point = cv2.minMaxLoc(probMap)

            # Scale the point to fit on the original image
            x = (self.width * point[0]) / W
            y = (self.height * point[1]) / H

            if prob > threshold:
                points.append((int(x), int(y)))

            else:
                points.append(None)

        # Always collect keypoints so the average-pose/trajectories images and the
        # keypoint cache (for fast re-rendering) are available; file-writing is gated below.
        time = frame2ms(ii, self.fps)
        points_list = [[list(point)[0]/self.width, list(point)[1]/self.height, ] if point is not None else [
            0, 0] for point in points]
        points_list_flat = itertools.chain.from_iterable(points_list)
        datapoint = [time]
        datapoint += points_list_flat
        data.append(datapoint)

        # Draw on the video frame, or on a plain canvas when overlay is disabled
        canvas, line_color, marker_color = _pose_canvas_and_colors(frame, overlay, background)

        # Marker history trails (last N frames)
        if _trail is not None:
            _trail.append([[p[0], p[1]] if p is not None else [np.nan, np.nan] for p in points])
            _draw_marker_trails(canvas, _trail, marker_color)

        # Joint lines (skeleton)
        if style in ('both', 'skeleton'):
            for pair in POSE_PAIRS:
                partA, partB = pair[0], pair[1]
                if points[partA] and points[partB]:
                    cv2.line(canvas, points[partA], points[partB],
                             line_color, 2, lineType=cv2.LINE_AA)

        # Markers (keypoints)
        if style in ('both', 'markers'):
            for point in points:
                if point is not None:
                    cv2.circle(canvas, point, 4, marker_color, thickness=-1, lineType=cv2.FILLED)

        frame = canvas

        if save_video:
            if video_out is None:
                assert target_name_video is not None  # resolved under `if save_video` above
                cmd =['ffmpeg', '-y', '-s', '{}x{}'.format(frame.shape[1], frame.shape[0]), 
                    '-r', str(self.fps), '-f', 'rawvideo', '-pix_fmt', 'bgr24', '-vcodec', 'rawvideo', 
                    '-i', '-', '-vcodec', 'libx264', '-pix_fmt', 'yuv420p', target_name_video]
                video_out = ffmpeg_cmd(cmd, total_time=self.length, pipe='write')

            video_out.stdin.write(frame.astype(np.uint8))

        # Flush the buffer
        process.stdout.flush()
        pb.progress(ii)
        ii += 1

    # Terminate the processes
    if save_video:
        # the writer opens on the first frame, so this only holds for a video that had frames
        assert video_out is not None, "no frames were written: the video decoded to nothing"
        video_out.stdin.close()
        video_out.wait()
        # Check if the original video fil has audio
        if self.has_audio:
            source_audio = extract_wav(of + fex)
            embed_audio_in_video(source_audio, target_name_video)
            os.remove(source_audio)

    process.terminate()

    def save_txt(of, width, height, model, data, data_format, target_name_data, overwrite):
        """
        Helper function to export pose estimation data as textfile(s).
        """
        def save_single_file(of, width, height, model, data, data_format, target_name_data, overwrite):
            """
            Helper function to export pose estimation data as a textfile using pandas.
            """

            coco_table = ['Nose', 'Neck', 'Right Shoulder', 'Right Elbow', 'Right Wrist', 'Left Shoulder', 'Left Elbow', 'Left Wrist', 'Right Hip',
                          'Right Knee', 'Right Ankle', 'Left Hip', 'Left Knee', 'Left Ankle', 'Right Eye', 'Left Eye', 'Right Ear', 'Left Ear']
            mpi_table = ['Head', 'Neck', 'Right Shoulder', 'Right Elbow', 'Right Wrist', 'Left Shoulder', 'Left Elbow',
                         'Left Wrist', 'Right Hip', 'Right Knee', 'Right Ankle', 'Left Hip', 'Left Knee', 'Left Ankle', 'Chest']
            body_25_table = ['Nose', 'Neck', 'Right Shoulder', 'Right Elbow', 'Right Wrist', 'Left Shoulder', 'Left Elbow', 'Left Wrist', 'Mid Hip', 'Right Hip', 'Right Knee', 'Right Ankle', 'Left Hip',
                             'Left Knee', 'Left Ankle', 'Right Eye', 'Left Eye', 'Right Ear', 'Left Ear', "Left Big Toe", "Left Small Toe", "Left Heel", "Right Big Toe", "Right Small Toe", "Right Heel"]
            headers = ['Time']

            table_to_use = []
            if model.lower() == 'mpi':
                table_to_use = mpi_table
            elif model.lower() == 'coco':
                table_to_use = coco_table
            elif model.lower() == 'body_25':
                table_to_use = body_25_table

            for i in range(len(table_to_use)):
                header_x = table_to_use[i] + ' X'
                header_y = table_to_use[i] + ' Y'
                headers.append(header_x)
                headers.append(header_y)

            data_format = data_format.lower()

            df = pd.DataFrame(data=data, columns=headers)

            if data_format == "tsv":

                if target_name_data is None:
                    target_name_data = of+'_pose.tsv'
                else:
                    # take name, but enforce tsv
                    target_name_data = os.path.splitext(
                        target_name_data)[0] + '.tsv'
                if not overwrite:
                    target_name_data = generate_outfilename(target_name_data)

                with open(target_name_data, 'wb') as f:
                    head_str = ''
                    for head in headers:
                        head_str += head + '\t'
                    head_str += '\n'
                    f.write(head_str.encode())
                    fmt_list = ['%d']
                    fmt_list += ['%.15f' for item in range(
                        len(table_to_use)*2)]
                    np.savetxt(f, df.values, delimiter='\t', fmt=fmt_list)

            elif data_format == "csv":

                if target_name_data is None:
                    target_name_data = of+'_pose.csv'
                else:
                    # take name, but enforce csv
                    target_name_data = os.path.splitext(
                        target_name_data)[0] + '.csv'
                if not overwrite:
                    target_name_data = generate_outfilename(target_name_data)

                df.to_csv(target_name_data, index=None)

            elif data_format == "txt":

                if target_name_data is None:
                    target_name_data = of+'_pose.txt'
                else:
                    # take name, but enforce txt
                    target_name_data = os.path.splitext(
                        target_name_data)[0] + '.txt'
                if not overwrite:
                    target_name_data = generate_outfilename(target_name_data)

                with open(target_name_data, 'wb') as f:
                    head_str = ''
                    for head in headers:
                        head_str += head + ' '
                    head_str += '\n'
                    f.write(head_str.encode())
                    fmt_list = ['%d']
                    fmt_list += ['%.15f' for item in range(
                        len(table_to_use)*2)]
                    np.savetxt(f, df.values, delimiter=' ', fmt=fmt_list)
            elif data_format not in ["tsv", "csv", "txt"]:
                print(
                    f"Invalid data format: '{data_format}'.\nFalling back to '.csv'.")
                save_single_file(of, width, height, model, data, "csv",
                                 target_name_data=target_name_data, overwrite=overwrite)

        if type(data_format) == str:
            save_single_file(of, width, height, model, data, data_format,
                             target_name_data=target_name_data, overwrite=overwrite)

        elif type(data_format) == list:
            if all([item.lower() in ["csv", "tsv", "txt"] for item in data_format]):
                data_format = list(set(data_format))
                [save_single_file(of, width, height, model, data, item, target_name_data=target_name_data, overwrite=overwrite)
                 for item in data_format]
            else:
                print(
                    f"Unsupported formats in {data_format}.\nFalling back to '.csv'.")
                save_single_file(of, width, height, model, data, "csv",
                                 target_name_data=target_name_data, overwrite=overwrite)

    if save_data:
        text_format = _handle_c3d(of, data, OPENPOSE_NAMES.get(model.lower()), self.fps,
                                  self.width, self.height, data_format, target_name_data, overwrite)
        if text_format is not None:
            save_txt(of, self.width, self.height, model, data, text_format,
                     target_name_data=target_name_data, overwrite=overwrite)

    # Render the average-pose and trajectories images from the collected keypoints
    names = OPENPOSE_NAMES.get(model.lower())
    # Cache keypoints so a later pose() call can re-render a different style without re-inference
    self._pose_keypoints = {
        'model': model.lower(), 'threshold': threshold, 'downsampling_factor': downsampling_factor,
        'names': names, 'connections': POSE_PAIRS, 'data': data,
        'width': self.width, 'height': self.height, 'fps': self.fps,
        'filename': filename, 'of': of, 'fex': fex, 'output_fex': output_fex,
        'has_audio': self.has_audio,
    }
    avg_frame = np.rint(avg_acc / avg_n).astype(np.uint8) if (avg_acc is not None and avg_n > 0) else None
    average_image, trajectories_image = _render_pose_extras(
        data, names, POSE_PAIRS, self.width, self.height, self.fps,
        avg_frame, of, save_average_pose, save_trajectories,
        target_name_average, target_name_trajectories, overwrite,
        transparent_trajectories=transparent_trajectories,
        trajectory_background=trajectory_background,
        trajectory_labels=trajectory_labels, style=style, background=background)
    self.pose_average_image = average_image
    self.pose_trajectories_image = trajectories_image

    if save_video:
        # save result as pose_video for parent MgVideo
        assert target_name_video is not None  # resolved under `if save_video` above
        self.pose_video = musicalgestures.MgVideo(target_name_video, color=self.color, returned_by_process=True)
        self.pose_video.average_pose = average_image
        self.pose_video.trajectories = trajectories_image
        return self.pose_video
    else:
        # otherwise just return the parent MgVideo
        return self

pose_waterfall

pose_waterfall(style='trajectories', n_samples=40, markers=None, color_by=None, cmap='hsv', dpi=200, elev=20, azim=-60, lw=1.0, axes=True, crop=False, target_name=None, overwrite=True, **pose_kwargs)

Render a 3D spatio-temporal waterfall of the pose, cascading along the time (depth) axis — a pose-based counterpart to silhouette_waterfall(). Uses cached pose keypoints from a previous pose() call when available; otherwise it runs pose estimation first (extra keyword arguments such as model/device/downsampling_factor are forwarded to pose()).

Parameters:

Name	Type	Description	Default
style	str	What to draw. 'trajectories' (default) draws each marker's continuous path through (x, time, y); 'markers' scatters the markers at n_samples time slices; 'skeleton' draws the skeleton joint lines at each time slice; 'both' draws markers + skeleton.	'trajectories'
n_samples	int	Number of time slices for the marker/skeleton styles. Defaults to 40 (ignored for 'trajectories').	40
markers	list	Subset of marker names or indices to draw. Defaults to all.	None
color_by	str	'marker' or 'time'. Defaults to None ("auto"): 'marker' for trajectories, 'time' for the slice styles.	None
cmap	str	Matplotlib colormap. Defaults to 'hsv'.	'hsv'
dpi	int	Output DPI. Defaults to 200.	200
elev	float	3D elevation angle. Defaults to 20.	20
azim	float	3D azimuth angle. Defaults to -60.	-60
lw	float	Line width. Defaults to 1.0.	1.0
axes	bool	Draw the axes and tick labels. Set to False for a clean render with all axes and text removed. Defaults to True.	True
crop	bool	Tighten the spatial limits to the marker extent and trim the surrounding whitespace, so the figure shows mostly the data. Defaults to False.	False
target_name	str	Output name. Defaults to None ("_pose_waterfall.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
**pose_kwargs		Forwarded to pose() if keypoints have to be computed.	{}

Returns:

Type	Description
'MgFigure | None'	MgFigure | None: the 3D waterfall figure, or None if there are too few frames.

Source code in musicalgestures/_pose.py

def mg_pose_waterfall(self: "musicalgestures.MgVideo", style: str = 'trajectories', n_samples: int = 40, markers: list | None = None, color_by: str | None = None,
                      cmap: str = 'hsv', dpi: int = 200, elev: float = 20, azim: float = -60, lw: float = 1.0, axes: bool = True, crop: bool = False,
                      target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure | None":
    """
    Render a 3D spatio-temporal waterfall of the pose, cascading along the time (depth) axis —
    a pose-based counterpart to ``silhouette_waterfall()``. Uses cached pose keypoints from a
    previous ``pose()`` call when available; otherwise it runs pose estimation first (extra
    keyword arguments such as ``model``/``device``/``downsampling_factor`` are forwarded to
    ``pose()``).

    Args:
        style (str, optional): What to draw. ``'trajectories'`` (default) draws each marker's
            continuous path through (x, time, y); ``'markers'`` scatters the markers at
            ``n_samples`` time slices; ``'skeleton'`` draws the skeleton joint lines at each
            time slice; ``'both'`` draws markers + skeleton.
        n_samples (int, optional): Number of time slices for the marker/skeleton styles.
            Defaults to 40 (ignored for ``'trajectories'``).
        markers (list, optional): Subset of marker names or indices to draw. Defaults to all.
        color_by (str, optional): ``'marker'`` or ``'time'``. Defaults to None ("auto"):
            'marker' for trajectories, 'time' for the slice styles.
        cmap (str, optional): Matplotlib colormap. Defaults to 'hsv'.
        dpi (int, optional): Output DPI. Defaults to 200.
        elev (float, optional): 3D elevation angle. Defaults to 20.
        azim (float, optional): 3D azimuth angle. Defaults to -60.
        lw (float, optional): Line width. Defaults to 1.0.
        axes (bool, optional): Draw the axes and tick labels. Set to False for a clean render
            with all axes and text removed. Defaults to True.
        crop (bool, optional): Tighten the spatial limits to the marker extent and trim the
            surrounding whitespace, so the figure shows mostly the data. Defaults to False.
        target_name (str, optional): Output name. Defaults to None ("_pose_waterfall.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        **pose_kwargs: Forwarded to ``pose()`` if keypoints have to be computed.

    Returns:
        MgFigure | None: the 3D waterfall figure, or None if there are too few frames.
    """
    from musicalgestures._pose_visualize import render_pose_waterfall

    _ensure_pose_keypoints(self, **pose_kwargs)

    c = self._pose_keypoints
    if target_name is None:
        target_name = c['of'] + '_pose_waterfall.png'
    else:
        target_name = os.path.splitext(target_name)[0] + '.png'

    mgf = render_pose_waterfall(
        c['data'], c['names'], c['width'], c['height'], c['fps'], target_name,
        overwrite=overwrite, style=style, connections=c.get('connections'),
        n_samples=n_samples, markers=markers, color_by=color_by, cmap=cmap,
        dpi=dpi, elev=elev, azim=azim, lw=lw, axes=axes, crop=crop)
    # a figure that could not be built is not a result: leaving the attribute
    # absent keeps `hasattr` honest, where storing None would not
    if mgf is not None:
        self.pose_waterfall_figure = mgf
    return mgf

pose_segments

pose_segments(segments=None, n_bins=36, cmap='viridis', dpi=200, ncols=6, target_name=None, overwrite=True, **pose_kwargs)

Circular (polar) motion plots and statistics for each body segment.

A segment is the bone between two connected joints (e.g. shoulder–elbow). For every segment this computes its per-frame orientation angle and draws a polar rose histogram of the angle distribution with the mean-direction resultant vector, annotated with circular statistics (mean angle, resultant length R, and range of motion). A CSV of the per-segment statistics — mean angle, R, circular std, range of motion, and mean angular speed — is saved alongside the image. Uses cached pose keypoints from a previous pose() call when available; otherwise it runs pose estimation first (model/device/… are forwarded to pose()).

Parameters:

Name	Type	Description	Default
segments	list	Subset of connections as (a, b) joint-index tuples. Defaults to all skeleton connections.	None
n_bins	int	Number of angular bins per rose. Defaults to 36 (10° bins).	36
cmap	str	Matplotlib colormap for the bars. Defaults to 'viridis'.	'viridis'
dpi	int	Output DPI. Defaults to 200.	200
ncols	int	Columns in the subplot grid. Defaults to 6.	6
target_name	str	Output name. Defaults to None ("_pose_segments.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
**pose_kwargs		Forwarded to pose() if keypoints have to be computed.	{}

Returns:

Type	Description
'MgFigure | None'	MgFigure | None: the grid of circular plots (per-segment stats in .data['stats']), or None.

Source code in musicalgestures/_pose.py

def mg_pose_segments(self: "musicalgestures.MgVideo", segments: list | None = None, n_bins: int = 36, cmap: str = 'viridis', dpi: int = 200, ncols: int = 6,
                     target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure | None":
    """
    Circular (polar) motion plots and statistics for each body segment.

    A *segment* is the bone between two connected joints (e.g. shoulder–elbow). For every segment
    this computes its per-frame orientation angle and draws a polar rose histogram of the angle
    distribution with the mean-direction resultant vector, annotated with circular statistics
    (mean angle, resultant length R, and range of motion). A CSV of the per-segment statistics —
    mean angle, R, circular std, range of motion, and mean angular speed — is saved alongside the
    image. Uses cached pose keypoints from a previous ``pose()`` call when available; otherwise it
    runs pose estimation first (``model``/``device``/… are forwarded to ``pose()``).

    Args:
        segments (list, optional): Subset of connections as ``(a, b)`` joint-index tuples.
            Defaults to all skeleton connections.
        n_bins (int, optional): Number of angular bins per rose. Defaults to 36 (10° bins).
        cmap (str, optional): Matplotlib colormap for the bars. Defaults to 'viridis'.
        dpi (int, optional): Output DPI. Defaults to 200.
        ncols (int, optional): Columns in the subplot grid. Defaults to 6.
        target_name (str, optional): Output name. Defaults to None ("_pose_segments.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        **pose_kwargs: Forwarded to ``pose()`` if keypoints have to be computed.

    Returns:
        MgFigure | None: the grid of circular plots (per-segment stats in ``.data['stats']``), or None.
    """
    from musicalgestures._pose_visualize import render_segment_circular

    _ensure_pose_keypoints(self, **pose_kwargs)

    c = self._pose_keypoints
    if target_name is None:
        target_name = c['of'] + '_pose_segments.png'
    else:
        target_name = os.path.splitext(target_name)[0] + '.png'

    mgf = render_segment_circular(
        c['data'], c['names'], c.get('connections'), c['width'], c['height'], c['fps'],
        target_name, overwrite=overwrite, segments=segments, n_bins=n_bins, cmap=cmap,
        dpi=dpi, ncols=ncols)
    # a figure that could not be built is not a result: leaving the attribute
    # absent keeps `hasattr` honest, where storing None would not
    if mgf is not None:
        self.pose_segments_figure = mgf
    return mgf

pose_center

pose_center(save_data=True, dpi=200, target_name=None, overwrite=True, **pose_kwargs)

Centre the pose data on its global centroid — a 2D port of the MoCap Toolbox mccenter.

A single offset per coordinate (the mean of the per-marker temporal means, missing detections ignored) is subtracted from every marker so the overall spatiotemporal centroid sits at the origin (0, 0). This removes the performer's absolute position in the frame, leaving relative posture/motion — useful before comparing or further analysing trajectories. Plots the centred marker trajectories and (by default) saves a CSV of the centred coordinates. Uses cached pose keypoints when available, otherwise runs pose() first (**pose_kwargs are forwarded).

Parameters:

Name	Type	Description	Default
save_data	bool	Save a CSV of the centred coordinates. Defaults to True.	True
dpi	int	Output DPI. Defaults to 200.	200
target_name	str	Output name. Defaults to None ("_pose_centered.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
**pose_kwargs		Forwarded to pose() if keypoints have to be computed.	{}

Returns:

Type	Description
'MgFigure | None'	MgFigure | None: the centred-trajectories figure; .data['coords'] holds the (T, n, 2) centred None when the figure could not be built, for instance with too few frames.
'MgFigure | None'	coordinates and .data['offset'] the removed centroid. None if there are too few frames.

Source code in musicalgestures/_pose.py

def mg_pose_center(self: "musicalgestures.MgVideo", save_data: bool = True, dpi: int = 200, target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure | None":
    """
    Centre the pose data on its global centroid — a 2D port of the MoCap Toolbox ``mccenter``.

    A single offset per coordinate (the mean of the per-marker temporal means, missing detections
    ignored) is subtracted from every marker so the overall spatiotemporal centroid sits at the
    origin (0, 0). This removes the performer's absolute position in the frame, leaving relative
    posture/motion — useful before comparing or further analysing trajectories. Plots the centred
    marker trajectories and (by default) saves a CSV of the centred coordinates. Uses cached pose
    keypoints when available, otherwise runs ``pose()`` first (``**pose_kwargs`` are forwarded).

    Args:
        save_data (bool, optional): Save a CSV of the centred coordinates. Defaults to True.
        dpi (int, optional): Output DPI. Defaults to 200.
        target_name (str, optional): Output name. Defaults to None ("_pose_centered.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        **pose_kwargs: Forwarded to ``pose()`` if keypoints have to be computed.

    Returns:
        MgFigure | None: the centred-trajectories figure; ``.data['coords']`` holds the (T, n, 2) centred
            None when the figure could not be built, for instance with too few frames.
        coordinates and ``.data['offset']`` the removed centroid. None if there are too few frames.
    """
    from musicalgestures._pose_visualize import render_pose_center

    _ensure_pose_keypoints(self, **pose_kwargs)

    c = self._pose_keypoints
    if target_name is None:
        target_name = c['of'] + '_pose_centered.png'
    else:
        target_name = os.path.splitext(target_name)[0] + '.png'

    mgf = render_pose_center(c['data'], c['names'], c['width'], c['height'], target_name,
                             overwrite=overwrite, dpi=dpi)
    if mgf is not None and save_data:
        try:
            import pandas as pd
            coords = mgf.data['coords']            # (T, n, 2)
            times = mgf.data['times']
            cols = {'Time': np.round(times * 1000.0, 3)}
            for i, name in enumerate(c['names']):
                cols[f'{name} X'] = coords[:, i, 0]
                cols[f'{name} Y'] = coords[:, i, 1]
            pd.DataFrame(cols).to_csv(os.path.splitext(target_name)[0] + '.csv', index=False)
        except Exception as e:
            print(f'Warning: could not save CSV: {e}')
    # a figure that could not be built is not a result: leaving the attribute
    # absent keeps `hasattr` honest, where storing None would not
    if mgf is not None:
        self.pose_centered_figure = mgf
    return mgf

pose_distance

pose_distance(dpi=200, target_name=None, overwrite=True, **pose_kwargs)

Per-marker distance travelled and the average across markers — a 2D port of the MoCap Toolbox mccumdist.

Sums each marker's frame-to-frame Euclidean displacement (in pixels) and accumulates it over time. The figure shows the per-marker cumulative-distance curves and a ranked bar chart of the total distance per marker with the across-marker average marked; a CSV of the totals (plus the average) is saved. Uses cached pose keypoints when available, otherwise runs pose() first (**pose_kwargs are forwarded).

Parameters:

Name	Type	Description	Default
dpi	int	Output DPI. Defaults to 200.	200
target_name	str	Output name. Defaults to None ("_pose_distance.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
**pose_kwargs		Forwarded to pose() if keypoints have to be computed.	{}

Returns:

Type	Description
'MgFigure | None'	MgFigure | None: .data['total'] (per-marker totals), .data['average'] (mean total), and None when the figure could not be built, for instance with too few frames.
'MgFigure | None'	.data['cumulative'] (per-marker cumulative curves). None if there are too few frames.

Source code in musicalgestures/_pose.py

def mg_pose_distance(self: "musicalgestures.MgVideo", dpi: int = 200, target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure | None":
    """
    Per-marker distance travelled and the average across markers — a 2D port of the MoCap Toolbox
    ``mccumdist``.

    Sums each marker's frame-to-frame Euclidean displacement (in pixels) and accumulates it over
    time. The figure shows the per-marker cumulative-distance curves and a ranked bar chart of the
    total distance per marker with the across-marker average marked; a CSV of the totals (plus the
    average) is saved. Uses cached pose keypoints when available, otherwise runs ``pose()`` first
    (``**pose_kwargs`` are forwarded).

    Args:
        dpi (int, optional): Output DPI. Defaults to 200.
        target_name (str, optional): Output name. Defaults to None ("_pose_distance.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        **pose_kwargs: Forwarded to ``pose()`` if keypoints have to be computed.

    Returns:
        MgFigure | None: ``.data['total']`` (per-marker totals), ``.data['average']`` (mean total), and
            None when the figure could not be built, for instance with too few frames.
        ``.data['cumulative']`` (per-marker cumulative curves). None if there are too few frames.
    """
    from musicalgestures._pose_visualize import render_pose_distance

    _ensure_pose_keypoints(self, **pose_kwargs)

    c = self._pose_keypoints
    if target_name is None:
        target_name = c['of'] + '_pose_distance.png'
    else:
        target_name = os.path.splitext(target_name)[0] + '.png'

    mgf = render_pose_distance(c['data'], c['names'], c['width'], c['height'], c['fps'],
                               target_name, overwrite=overwrite, dpi=dpi)
    # a figure that could not be built is not a result: leaving the attribute
    # absent keeps `hasattr` honest, where storing None would not
    if mgf is not None:
        self.pose_distance_figure = mgf
    return mgf

painting

painting(reference_s=5.0, width=200, dpi=110, title=None, save_data=True, target_name=None, target_name_data=None, target_name_colourgram=None, overwrite=True)

Measure the painting in this video (the video should frame the canvas) and draw it.

Parameters:

Name	Type	Description	Default
reference_s	float	Seconds at the start taken as the initial canvas. Defaults to 5.	5.0
width	int	Working width in pixels. Defaults to 200.	200
dpi	int	Figure resolution.	110
title	str	Figure title.	None
save_data	bool	Write the per-second table as CSV. Defaults to True.	True
target_name	str	Figure path. Defaults to "_painting.png" beside the video.	None
target_name_data	str	CSV path. Defaults to "_painting.csv".	None
target_name_colourgram	str	Raw colourgram path. Defaults to "_colourgram.png".	None
overwrite	bool	Overwrite or auto-increment. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	MgFigure	With the content dict in .data; also stored as self.painting_figure,
	MgFigure	the colourgram as self.colourgram_image and the table as self.painting.

Source code in musicalgestures/_canvas.py

def mg_painting(self, reference_s: float = 5.0, width: int = 200, dpi: int = 110, title: str | None = None,
                save_data: bool = True, target_name: str | None = None, target_name_data: str | None = None,
                target_name_colourgram: str | None = None, overwrite: bool = True) -> MgFigure:
    """Measure the painting in this video (the video should frame the canvas) and draw it.

    Args:
        reference_s (float): Seconds at the start taken as the initial canvas. Defaults to 5.
        width (int): Working width in pixels. Defaults to 200.
        dpi (int): Figure resolution.
        title (str, optional): Figure title.
        save_data (bool): Write the per-second table as CSV. Defaults to True.
        target_name (str, optional): Figure path. Defaults to ``"_painting.png"`` beside the video.
        target_name_data (str, optional): CSV path. Defaults to ``"_painting.csv"``.
        target_name_colourgram (str, optional): Raw colourgram path. Defaults to ``"_colourgram.png"``.
        overwrite (bool): Overwrite or auto-increment. Defaults to True.

    Returns:
        MgFigure: With the content dict in ``.data``; also stored as ``self.painting_figure``,
        the colourgram as ``self.colourgram_image`` and the table as ``self.painting``.
    """
    import matplotlib
    import matplotlib.pyplot as plt
    content = painting_content(self.filename, reference_s=reference_s, width=width)
    t = content["t"]
    img = colourgram_image(content)
    cg = resolve_filename(self.of, "_colourgram.png", target_name_colourgram, overwrite)
    matplotlib.image.imsave(cg, img)
    self.colourgram_image = MgImage(cg)
    fig, axs = plt.subplots(3, 1, figsize=(16, 8), sharex=True, dpi=dpi, gridspec_kw={"height_ratios": [2, 1.2, 1.2]})
    axs[0].imshow(img, aspect="auto", extent=[0, t[-1] + 0.5, 0, 360], interpolation="nearest")
    axs[0].set_ylabel("hue (°)"); axs[0].set_title(title or f"{os.path.basename(self.of)}: colourgram and painting content")
    axs[1].plot(t, content["coverage"], lw=1.5, label="coverage (monotone)"); axs[1].plot(t, content["painted"], lw=.5, alpha=.4, label="painted share")
    axs[1].plot(t, content["chromatic"], lw=1.2, label="chromatic share"); axs[1].plot(t, content["edge_density"] * 5, lw=1.2, label="edge density (x5)")
    axs[1].set_ylim(0, 1); axs[1].legend(fontsize=8, ncol=4); axs[1].set_ylabel("fraction")
    axs[2].plot(t, content["warm"], lw=1.2, color="#D9534F", label="warm hues"); axs[2].plot(t, content["cool"], lw=1.2, color="#3B6FB6", label="cool hues")
    axs[2].plot(t, content["saturation"], lw=1, color="grey", label="mean saturation"); axs[2].plot(t, content["composition"]["mass_y"], lw=1, color="black", ls="--", label="paint centre (y, 0 = top)")
    axs[2].legend(fontsize=8, ncol=4); axs[2].set_ylabel("fraction"); axs[2].set_xlabel("time (s)")
    fig.tight_layout()
    path = resolve_filename(self.of, "_painting.png", target_name, overwrite)
    fig.savefig(path); plt.close(fig)
    if save_data:
        import pandas as pd
        df = pd.DataFrame({"time": t, **{k: content[k] for k in ("painted", "coverage", "chromatic", "saturation", "brightness", "warm", "cool", "edge_density")},
                           **{k: content["composition"][k] for k in ("mass_x", "mass_y", "spread_x", "spread_y", "symmetry_lr", "anisotropy")}})
        for k in range(content["hue_hist"].shape[0]):
            df[f"hue_{int(content['hue_centres'][k] - 5):03d}"] = content["hue_hist"][k]
        dp = resolve_filename(self.of, "_painting.csv", target_name_data, overwrite)
        df.to_csv(dp, index=False, float_format="%.4f")
        self.painting = df
    result = MgFigure(figure=fig, figure_type="video.painting", data={k: v for k, v in content.items() if k != "frames"}, layers=None, image=path)
    self.painting_figure = result
    return result

eyetracking

eyetracking(export_dir, start=0.0, blink_pad_frames=3, save_data=True, target_name=None, overwrite=True)

Align a Pupil Labs export to this video and keep the frame table on it.

Parameters:

Name	Type	Description	Default
export_dir	str | Path	The Pupil Cloud export folder.	required
start	str | float	Where this video begins on the recording clock: an event name from events.csv ("Music begins") or seconds after the recording start. Defaults to 0.0.	0.0
blink_pad_frames	int	See :func:pupil_to_frames. Defaults to 3.	3
save_data	bool	Write the table as CSV beside the video. Defaults to True.	True
target_name	str	Output path. Defaults to "_eyetracking.csv" beside the video.	None
overwrite	bool	Overwrite or auto-increment. Defaults to True.	True

Returns:

Type	Description
DataFrame	pandas.DataFrame: The frame table, also stored as self.eyetracking; the
DataFrame	recording as self.pupil_recording.

Source code in musicalgestures/_pupillabs.py

def mg_eyetracking(self, export_dir, start=0.0, blink_pad_frames: int = 3,
                   save_data: bool = True, target_name: str | None = None,
                   overwrite: bool = True) -> pd.DataFrame:
    """Align a Pupil Labs export to this video and keep the frame table on it.

    Args:
        export_dir (str | Path): The Pupil Cloud export folder.
        start (str | float): Where this video begins on the recording clock: an event
            name from events.csv (``"Music begins"``) or seconds after the recording
            start. Defaults to 0.0.
        blink_pad_frames (int): See :func:`pupil_to_frames`. Defaults to 3.
        save_data (bool): Write the table as CSV beside the video. Defaults to True.
        target_name (str, optional): Output path. Defaults to ``"_eyetracking.csv"``
            beside the video.
        overwrite (bool): Overwrite or auto-increment. Defaults to True.

    Returns:
        pandas.DataFrame: The frame table, also stored as `self.eyetracking`; the
        recording as `self.pupil_recording`.
    """
    rec = read_pupil_export(export_dir)
    frames = pupil_to_frames(rec, float(self.fps), start=start, duration_s=float(self.duration),
                             blink_pad_frames=blink_pad_frames)
    self.pupil_recording = rec
    self.eyetracking = frames
    if save_data:
        path = resolve_filename(self.of, "_eyetracking.csv", target_name, overwrite)
        frames.to_csv(path, index=False, float_format="%.4f")
        self.eyetracking_data = path
    return frames

gazegrams

gazegrams(bins=120, bin_s=1.0, colormap='magma', target_name_x=None, target_name_y=None, overwrite=True)

Write the horizontal and vertical gazegrams of an aligned export as images.

Oriented like the motiongrams: the _ggy image has time across and vertical scene position down; the _ggx image has time down and horizontal position across.

Parameters:

Name	Type	Description	Default
bins	int	Position bins. Defaults to 120.	120
bin_s	float	Time bin in seconds. Defaults to 1.0.	1.0
colormap	str	Matplotlib colormap. Defaults to "magma".	'magma'
target_name_x	str	Path for the horizontal-position gazegram.	None
target_name_y	str	Path for the vertical-position gazegram.	None
overwrite	bool	Overwrite or auto-increment. Defaults to True.	True

Returns:

Name	Type	Description
MgList	MgList	MgImage for the x and y gazegrams, also stored as self.gazegram_x_image
	MgList	and self.gazegram_y_image.

Source code in musicalgestures/_pupillabs.py

def mg_gazegrams(self, bins: int = 120, bin_s: float = 1.0, colormap: str = "magma",
                 target_name_x: str | None = None, target_name_y: str | None = None,
                 overwrite: bool = True) -> MgList:
    """Write the horizontal and vertical gazegrams of an aligned export as images.

    Oriented like the motiongrams: the `_ggy` image has time across and vertical scene
    position down; the `_ggx` image has time down and horizontal position across.

    Args:
        bins (int): Position bins. Defaults to 120.
        bin_s (float): Time bin in seconds. Defaults to 1.0.
        colormap (str): Matplotlib colormap. Defaults to ``"magma"``.
        target_name_x (str, optional): Path for the horizontal-position gazegram.
        target_name_y (str, optional): Path for the vertical-position gazegram.
        overwrite (bool): Overwrite or auto-increment. Defaults to True.

    Returns:
        MgList: `MgImage` for the x and y gazegrams, also stored as `self.gazegram_x_image`
        and `self.gazegram_y_image`.
    """
    import matplotlib
    frames = _require_frames(self)
    rec = getattr(self, "pupil_recording", None)
    w, h = rec.scene_size if rec is not None else DEFAULT_SCENE_SIZE
    cmap = matplotlib.colormaps[colormap]
    gy = gazegram(frames, "y", h, bin_s, bins)
    gx = gazegram(frames, "x", w, bin_s, bins)
    py = resolve_filename(self.of, "_ggy.png", target_name_y, overwrite)
    px = resolve_filename(self.of, "_ggx.png", target_name_x, overwrite)
    matplotlib.image.imsave(py, gy, cmap=cmap)
    matplotlib.image.imsave(px, gx.T, cmap=cmap)
    self.gazegram_y_image = MgImage(py)
    self.gazegram_x_image = MgImage(px)
    return MgList(self.gazegram_x_image, self.gazegram_y_image)

eyetracking_timeline

eyetracking_timeline(bin_s=1.0, dpi=120, title=None, target_name=None, overwrite=True)

Draw gaze velocity, event rates, pupil size and head rotation over the video.

Parameters:

Name	Type	Description	Default
bin_s	float	Bin for the rates. Defaults to 1.0.	1.0
dpi	int	Figure resolution. Defaults to 120.	120
title	str	Figure title. Defaults to the video's name.	None
target_name	str	Output path. Defaults to "_eyetracking.png".	None
overwrite	bool	Overwrite or auto-increment. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	MgFigure	With the per-bin table in .data["rates"]; also stored as
	MgFigure	self.eyetracking_figure.

Source code in musicalgestures/_pupillabs.py

def mg_eyetracking_timeline(self, bin_s: float = 1.0, dpi: int = 120, title: str | None = None,
                            target_name: str | None = None, overwrite: bool = True) -> MgFigure:
    """Draw gaze velocity, event rates, pupil size and head rotation over the video.

    Args:
        bin_s (float): Bin for the rates. Defaults to 1.0.
        dpi (int): Figure resolution. Defaults to 120.
        title (str, optional): Figure title. Defaults to the video's name.
        target_name (str, optional): Output path. Defaults to ``"_eyetracking.png"``.
        overwrite (bool): Overwrite or auto-increment. Defaults to True.

    Returns:
        MgFigure: With the per-bin table in `.data["rates"]`; also stored as
        `self.eyetracking_figure`.
    """
    import matplotlib.pyplot as plt
    frames = _require_frames(self)
    rates = eyetracking_rates(frames, float(self.fps), bin_s)
    panels = [c for c in ("gaze_vel", "fixation_rate", "blink_rate", "pupil_mean", "head_gyro") if c in rates]
    fig, axs = plt.subplots(len(panels), 1, figsize=(16, 1.8 * len(panels) + 1), sharex=True, squeeze=False, dpi=dpi)
    labels = {"gaze_vel": "gaze velocity (deg/s)", "fixation_rate": "fixations / s", "blink_rate": "blinks / s",
              "pupil_mean": "pupil (mm)", "head_gyro": "head rotation (deg/s)"}
    for ax, c in zip(axs[:, 0], panels):
        ax.plot(rates.index.values, rates[c].values, lw=0.7)
        ax.set_ylabel(labels[c], fontsize=8)
    axs[-1, 0].set_xlabel("time (s)")
    axs[0, 0].set_title(title or f"{os.path.basename(self.of)}: eye tracking and head motion")
    fig.tight_layout()
    path = resolve_filename(self.of, "_eyetracking.png", target_name, overwrite)
    fig.savefig(path)
    plt.close(fig)
    result = MgFigure(figure=fig, figure_type="video.eyetracking",
                      data={"rates": rates}, layers=None, image=path)
    self.eyetracking_figure = result
    return result

posegram

posegram(landmarks=None, times=None, colormap='magma', gamma=0.5, max_width=4000, dpi=130, target_name=None, overwrite=True)

Draw the posegram: landmarks head to foot down the page, time across it.

Parameters:

Name	Type	Description	Default
landmarks	ndarray	(frames, 33, 3) from a previous extract_pose_landmarks call. Defaults to None, which runs pose here --- expensive on a long recording, so pass a saved extraction when you have one.	None
times	array - like	Seconds per frame, needed whenever the landmarks were not sampled at the video's own frame rate. Pose is usually extracted at a reduced rate, so this is usually needed.	None
colormap	str	Defaults to 'magma'.	'magma'
gamma	float	Applied before colouring so quiet passages stay visible. Defaults to 0.5.	0.5
max_width	int	Widest the drawn image may be. One column per frame makes a 149-megapixel picture of a long session; columns are pooled by maximum above this, so a brief accent still shows. Defaults to 4000.	4000
dpi	int	Defaults to 130.	130
target_name	str	Output path. Defaults to the input name with _posegram.	None
overwrite	bool	Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'musicalgestures.MgFigure'	the posegram, with the activity array in .data.

Source code in musicalgestures/_posegram.py

def mg_posegram(self: "musicalgestures.MgVideo", landmarks=None, times=None,
                colormap: str = "magma", gamma: float = 0.5, max_width: int = 4000,
                dpi: int = 130, target_name: str | None = None,
                overwrite: bool = True) -> "musicalgestures.MgFigure":
    """Draw the posegram: landmarks head to foot down the page, time across it.

    Args:
        landmarks (np.ndarray, optional): `(frames, 33, 3)` from a previous
            `extract_pose_landmarks` call. Defaults to None, which runs pose here ---
            expensive on a long recording, so pass a saved extraction when you have one.
        times (array-like, optional): Seconds per frame, needed whenever the landmarks
            were not sampled at the video's own frame rate. Pose is usually extracted at
            a reduced rate, so this is usually needed.
        colormap (str, optional): Defaults to `'magma'`.
        gamma (float, optional): Applied before colouring so quiet passages stay visible.
            Defaults to 0.5.
        max_width (int, optional): Widest the drawn image may be. One column per frame
            makes a 149-megapixel picture of a long session; columns are pooled by
            maximum above this, so a brief accent still shows. Defaults to 4000.
        dpi (int, optional): Defaults to 130.
        target_name (str, optional): Output path. Defaults to the input name with
            `_posegram`.
        overwrite (bool, optional): Defaults to True.

    Returns:
        MgFigure: the posegram, with the activity array in `.data`.
    """
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._utils import MgFigure

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_posegram.png', target_name, overwrite)

    if landmarks is None:
        from musicalgestures._posetools import extract_pose_landmarks
        r = extract_pose_landmarks(self.filename, quiet=True, verbose=False)
        landmarks, times = r["landmarks"], r["time"]
    activity = pose_activity(landmarks, anatomical=True)
    times = (np.asarray(times, dtype=np.float64) if times is not None
             else np.arange(activity.shape[1]) / float(self.fps))

    drawn = activity
    if drawn.shape[1] > max_width:
        edges = np.linspace(0, drawn.shape[1], max_width + 1).astype(int)
        drawn = np.stack([drawn[:, a:max(b, a + 1)].max(axis=1)
                          for a, b in zip(edges[:-1], edges[1:])], axis=1)
    ceiling = np.percentile(drawn[drawn > 0], 99) if (drawn > 0).any() else 1.0
    shown = np.power(np.clip(drawn / max(ceiling, 1e-9), 0, 1), gamma)

    minutes = (times[-1] / 60) if len(times) else 1
    fig, ax = plt.subplots(figsize=(15, 6), dpi=dpi)
    ax.imshow(shown, cmap=colormap, aspect="auto", interpolation="nearest",
              extent=(0, minutes, len(ANATOMICAL_ORDER), 0))
    seen = set()
    for label, upto in BANDS:
        if label in seen:
            continue
        seen.add(label)
        ax.axhline(upto, color="white", linewidth=0.6, alpha=0.35)
    ax.set_yticks(BAND_TICKS)
    ax.set_yticklabels([label for label, _ in BANDS], fontsize=9)
    ax.set_xlabel("minutes")
    ax.set_title(f"{os.path.basename(self.filename)} --- posegram "
                 f"(landmark speed, head to foot)", fontsize=10)
    fig.tight_layout()
    fig.savefig(target_name)
    plt.close(fig)

    self.posegram_figure = MgFigure(
        figure=None, figure_type="video.posegram",
        data={"activity": activity, "time": times, "order": ANATOMICAL_ORDER},
        layers=None, image=target_name)
    return self.posegram_figure

posegram_spatial

posegram_spatial(landmarks=None, times=None, frame_size=None, axis='vertical', weight='speed', bins=200, colormap='magma', gamma=0.5, max_width=4000, dpi=130, target_name=None, overwrite=True)

The posegram drawn on the image's axes, so it lines up with a motiongram.

posegram() puts one landmark per row, which answers "which body part moved". This puts image position on the vertical axis instead, which is what a motiongram does, so a body crossing the frame draws the same diagonal in both and the two can be laid against each other. Where they disagree, the pixels saw something the pose model did not, or the other way round.

Parameters:

Name	Type	Description	Default
landmarks	ndarray	(frames, 33, 3) in pixel coordinates from a previous extraction. Defaults to None, which runs pose here.	None
times	array - like	Seconds per frame. Pose is usually extracted at a reduced rate, so this is usually needed.	None
frame_size	tuple	(width, height) the landmarks were extracted in. Pass this whenever you pass landmarks. MediaPipe estimates landmarks it cannot see and puts them outside the picture --- on a real 640x360 extraction the largest y was 1529, four times the frame --- so inferring the frame from the data squeezes the whole body into a corner of the plot. Without it a high percentile is used, which is robust but still a guess.	None
axis	str	'vertical' bins by y, to match a vertical motiongram; 'horizontal' bins by x. Defaults to 'vertical'.	'vertical'
weight	str	'speed' for motion at each height, the motiongram's own quantity; 'presence' for where the body is regardless of motion. Defaults to 'speed'.	'speed'
bins	int	Rows. Defaults to 200.	200
colormap, gamma, max_width, dpi, target_name, overwrite		as posegram().	required

Returns:

Name	Type	Description
MgFigure	'musicalgestures.MgFigure'	the gram, with the array in .data.

Source code in musicalgestures/_posegram.py

def mg_posegram_spatial(self: "musicalgestures.MgVideo", landmarks=None, times=None,
                        frame_size=None,
                        axis: str = "vertical", weight: str = "speed", bins: int = 200,
                        colormap: str = "magma", gamma: float = 0.5,
                        max_width: int = 4000, dpi: int = 130,
                        target_name: str | None = None,
                        overwrite: bool = True) -> "musicalgestures.MgFigure":
    """The posegram drawn on the image's axes, so it lines up with a motiongram.

    `posegram()` puts one landmark per row, which answers "which body part moved". This
    puts **image position** on the vertical axis instead, which is what a motiongram does,
    so a body crossing the frame draws the same diagonal in both and the two can be laid
    against each other. Where they disagree, the pixels saw something the pose model did
    not, or the other way round.

    Args:
        landmarks (np.ndarray, optional): `(frames, 33, 3)` in pixel coordinates from a
            previous extraction. Defaults to None, which runs pose here.
        times (array-like, optional): Seconds per frame. Pose is usually extracted at a
            reduced rate, so this is usually needed.
        frame_size (tuple, optional): `(width, height)` the landmarks were extracted in.
            **Pass this whenever you pass `landmarks`.** MediaPipe estimates landmarks it
            cannot see and puts them outside the picture --- on a real 640x360 extraction
            the largest y was 1529, four times the frame --- so inferring the frame from
            the data squeezes the whole body into a corner of the plot. Without it a high
            percentile is used, which is robust but still a guess.
        axis (str, optional): `'vertical'` bins by y, to match a vertical motiongram;
            `'horizontal'` bins by x. Defaults to `'vertical'`.
        weight (str, optional): `'speed'` for motion at each height, the motiongram's own
            quantity; `'presence'` for where the body is regardless of motion. Defaults
            to `'speed'`.
        bins (int, optional): Rows. Defaults to 200.
        colormap, gamma, max_width, dpi, target_name, overwrite: as `posegram()`.

    Returns:
        MgFigure: the gram, with the array in `.data`.
    """
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._utils import MgFigure

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_posegram_spatial.png', target_name, overwrite)

    if landmarks is None:
        from musicalgestures._posetools import extract_pose_landmarks
        r = extract_pose_landmarks(self.filename, quiet=True, verbose=False)
        landmarks, times = r["landmarks"], r["time"]
        height, width = r["height"], r["width"]
    elif frame_size is not None:
        width, height = float(frame_size[0]), float(frame_size[1])
    else:
        #: No frame given, so it has to be inferred --- and NOT from the maximum, which is
        #: an estimated landmark somewhere outside the picture. A high percentile tracks
        #: the body and ignores the extrapolations.
        a = np.asarray(landmarks, dtype=np.float64)
        if np.isfinite(a).any():
            height = float(np.nanpercentile(a[:, :, 1], 99.5)) * 1.05
            width = float(np.nanpercentile(a[:, :, 0], 99.5)) * 1.05
        else:
            height = width = 1.0

    gram = pose_spatial_gram(landmarks, height=height, width=width, bins=bins,
                             axis=axis, weight=weight)
    times = (np.asarray(times, dtype=np.float64) if times is not None
             else np.arange(gram.shape[1]) / float(self.fps))

    drawn = gram
    if drawn.shape[1] > max_width:
        edges = np.linspace(0, drawn.shape[1], max_width + 1).astype(int)
        drawn = np.stack([drawn[:, a:max(b, a + 1)].max(axis=1)
                          for a, b in zip(edges[:-1], edges[1:])], axis=1)
    ceiling = np.percentile(drawn[drawn > 0], 99) if (drawn > 0).any() else 1.0
    shown = np.power(np.clip(drawn / max(ceiling, 1e-9), 0, 1), gamma)

    minutes = (times[-1] / 60) if len(times) else 1
    fig, ax = plt.subplots(figsize=(15, 5.5), dpi=dpi)
    ax.imshow(shown, cmap=colormap, aspect="auto", interpolation="nearest",
              extent=(0, minutes, bins, 0))
    ax.set_xlabel("minutes")
    ax.set_ylabel("top of frame  →  bottom" if axis == "vertical"
                  else "left of frame  →  right")
    ax.set_yticks([])
    ax.set_title(f"{os.path.basename(self.filename)} --- pose {axis} gram "
                 f"({weight}, on the image's own axis)", fontsize=10)
    fig.tight_layout()
    fig.savefig(target_name)
    plt.close(fig)

    self.posegram_spatial_figure = MgFigure(
        figure=None, figure_type="video.posegram_spatial",
        data={"gram": gram, "time": times, "axis": axis, "weight": weight},
        layers=None, image=target_name)
    return self.posegram_spatial_figure

posegrams

posegrams(landmarks=None, times=None, frame_size=None, bins=200, weight='speed', colormap='magma', gamma=0.5, max_width=4000, dpi=130, target_name=None, overwrite=True)

Posegrams of where the body actually was, in the frame's own coordinates.

The pose counterpart of motiongrams(), and oriented the same way: the horizontal view has time running across with image y down the page, the vertical view has time running down with image x across. Laid around a video frame they line up with it, and laid beside a motiongram of the same recording they can be read against it — a body crossing the room draws the same diagonal in both.

Because pose gives an actual position rather than a region of changed pixels, these are the true location over time, not an estimate of where change happened.

Parameters:

Name	Type	Description	Default
landmarks	ndarray	(frames, 33, 3) from a previous extraction. Defaults to None, which runs pose here.	None
times	array - like	Seconds per frame; usually needed, since pose is normally extracted at a reduced rate.	None
frame_size	tuple	(width, height) the landmarks were extracted in. Pass this whenever you pass landmarks --- see this module's notes on MediaPipe placing unseen landmarks outside the picture.	None
bins	int	Cells along the spatial axis. Defaults to 200.	200
weight	str	'speed' or 'presence'. Defaults to 'speed'.	'speed'
colormap, gamma, max_width, dpi, target_name, overwrite		as elsewhere.	required

Returns:

Name	Type	Description
MgList	'musicalgestures.MgList'	the horizontal and vertical posegrams, in that order.

Source code in musicalgestures/_posegram.py

def mg_posegrams(self: "musicalgestures.MgVideo", landmarks=None, times=None,
                 frame_size=None, bins: int = 200, weight: str = "speed",
                 colormap: str = "magma", gamma: float = 0.5, max_width: int = 4000,
                 dpi: int = 130, target_name: str | None = None,
                 overwrite: bool = True) -> "musicalgestures.MgList":
    """Posegrams of where the body actually was, in the frame's own coordinates.

    The pose counterpart of `motiongrams()`, and oriented the same way: the horizontal
    view has time running across with image y down the page, the vertical view has time
    running down with image x across. Laid around a video frame they line up with it, and
    laid beside a motiongram of the same recording they can be read against it — a body
    crossing the room draws the same diagonal in both.

    Because pose gives an actual position rather than a region of changed pixels, these
    are the *true location* over time, not an estimate of where change happened.

    Args:
        landmarks (np.ndarray, optional): `(frames, 33, 3)` from a previous extraction.
            Defaults to None, which runs pose here.
        times (array-like, optional): Seconds per frame; usually needed, since pose is
            normally extracted at a reduced rate.
        frame_size (tuple, optional): `(width, height)` the landmarks were extracted in.
            **Pass this whenever you pass `landmarks`** --- see this module's notes on
            MediaPipe placing unseen landmarks outside the picture.
        bins (int, optional): Cells along the spatial axis. Defaults to 200.
        weight (str, optional): `'speed'` or `'presence'`. Defaults to `'speed'`.
        colormap, gamma, max_width, dpi, target_name, overwrite: as elsewhere.

    Returns:
        MgList: the horizontal and vertical posegrams, in that order.
    """
    import matplotlib.pyplot as plt
    import numpy as np

    from musicalgestures._mglist import MgList
    from musicalgestures._utils import MgImage

    of, fex = os.path.splitext(self.filename)
    base = resolve_filename(of, '_posegram.png', target_name, overwrite)
    stem, _ = os.path.splitext(base)

    if landmarks is None:
        from musicalgestures._posetools import extract_pose_landmarks
        r = extract_pose_landmarks(self.filename, quiet=True, verbose=False)
        landmarks, times = r["landmarks"], r["time"]
        width, height = r["width"], r["height"]
    elif frame_size is not None:
        width, height = float(frame_size[0]), float(frame_size[1])
    else:
        a = np.asarray(landmarks, dtype=np.float64)
        width = float(np.nanpercentile(a[:, :, 0], 99.5)) * 1.05
        height = float(np.nanpercentile(a[:, :, 1], 99.5)) * 1.05

    horizontal, vertical = posegram_arrays(landmarks, width=width, height=height,
                                           bins=bins, weight=weight)
    times = (np.asarray(times, dtype=np.float64) if times is not None
             else np.arange(horizontal.shape[1]) / float(self.fps))
    minutes = (times[-1] / 60) if len(times) else 1

    def _shade(a):
        finite = a[np.isfinite(a) & (a > 0)]
        ceiling = np.percentile(finite, 99) if finite.size else 1.0
        return np.power(np.clip(a / max(ceiling, 1e-9), 0, 1), gamma)

    images = []
    for gram, suffix, horizontal_time in ((horizontal, "_h", True),
                                          (vertical, "_v", False)):
        drawn = gram
        axis = 1 if horizontal_time else 0
        if drawn.shape[axis] > max_width:
            edges = np.linspace(0, drawn.shape[axis], max_width + 1).astype(int)
            if horizontal_time:
                drawn = np.stack([drawn[:, a:max(b, a + 1)].max(axis=1)
                                  for a, b in zip(edges[:-1], edges[1:])], axis=1)
            else:
                drawn = np.stack([drawn[a:max(b, a + 1)].max(axis=0)
                                  for a, b in zip(edges[:-1], edges[1:])])
        if horizontal_time:
            fig, ax = plt.subplots(figsize=(15, 5), dpi=dpi)
            ax.imshow(_shade(drawn), cmap=colormap, aspect="auto",
                      interpolation="nearest", extent=(0, minutes, bins, 0))
            ax.set_xlabel("minutes")
            ax.set_ylabel("top of frame  →  bottom")
            ax.set_title("horizontal posegram — time across, image y down")
        else:
            fig, ax = plt.subplots(figsize=(6.5, 12), dpi=dpi)
            ax.imshow(_shade(drawn), cmap=colormap, aspect="auto",
                      interpolation="nearest", extent=(0, bins, minutes, 0))
            ax.set_ylabel("minutes")
            ax.set_xlabel("left of frame  →  right")
            ax.set_title("vertical posegram — time down, image x across")
        ax.set_xticks(ax.get_xticks()) if False else None
        fig.tight_layout()
        path = f"{stem}{suffix}.png"
        fig.savefig(path)
        plt.close(fig)
        images.append(MgImage(path))

    self.posegrams_images = MgList(images)
    return self.posegrams_images

__repr__

__repr__()

Source code in musicalgestures/_video.py

def __repr__(self) -> str:
    w, h = getattr(self, 'width', None), getattr(self, 'height', None)
    size = f"{w}x{h}" if w and h else "?x?"
    fps = getattr(self, 'fps', None)
    fps_str = f"{fps:g}fps" if fps else "?fps"
    frames = getattr(self, 'length', None)
    frames_str = f"{int(frames)} frames" if frames else "? frames"
    return (f"MgVideo('{self.filename}', {frames_str}, {fps_str}, {size}, "
            f"audio={getattr(self, 'has_audio', None)})")

average

average(**kwargs)

Backward compatibility alias for blend(component_mode='average'). Creates an average image of all frames in the video.

Parameters:

Name	Type	Description	Default
**kwargs		Additional arguments passed to blend method. Note: 'normalize' parameter is accepted for backward compatibility but ignored.	{}

Returns:

Name	Type	Description
MgImage		A new MgImage pointing to the output average image file.

Source code in musicalgestures/_video.py

def average(self, **kwargs):
    """
    Backward compatibility alias for blend(component_mode='average').
    Creates an average image of all frames in the video.

    Args:
        **kwargs: Additional arguments passed to blend method.
                 Note: 'normalize' parameter is accepted for backward compatibility but ignored.

    Returns:
        MgImage: A new MgImage pointing to the output average image file.
    """
    # Strip parameters that were documented in older API versions but aren't
    # supported by the underlying blend implementation.
    filtered_kwargs = {k: v for k, v in kwargs.items() if k not in ('normalize', 'method')}
    return self.blend(component_mode='average', **filtered_kwargs)

test_input

test_input()

Gives feedback to user if initialization from input went wrong.

Source code in musicalgestures/_video.py

def test_input(self):
    """Gives feedback to user if initialization from input went wrong."""
    mg_input_test(
        self.filename,
        self.array,
        self.fps,
        self.filtertype,
        self.threshold,
        self.starttime,
        self.endtime,
        self.blur,
        self.skip,
        self.frames,
    )

get_video

get_video()

Creates a video attribute to the Musical Gestures object with the given correct settings.

NB: For an MgVideo, self.length is the number of frames (from get_framecount), whereas for MgAudio self.length is the duration in seconds. To get the video duration in seconds use self.length / self.fps.

Source code in musicalgestures/_video.py

def get_video(self):
    """Creates a video attribute to the Musical Gestures object with the given correct settings.

    NB: For an ``MgVideo``, ``self.length`` is the number of **frames** (from
    ``get_framecount``), whereas for ``MgAudio`` ``self.length`` is the duration in
    **seconds**. To get the video duration in seconds use ``self.length / self.fps``.
    """
    # Bake any display-rotation flag into the pixels so every reader (FFmpeg pipe,
    # OpenCV, filters) agrees on orientation and no process comes out rotated.
    from musicalgestures._utils import normalize_rotation
    oriented = normalize_rotation(self.filename)
    if oriented != self.filename:
        self.filename = oriented
        self.of, self.fex = os.path.splitext(self.filename)

    (
        self.length,
        self.width,
        self.height,
        self.fps,
        self.endtime,
        self.of,
        self.fex,
        self.has_audio,
    ) = mg_videoreader(
        filename=self.filename,
        starttime=self.starttime,
        endtime=self.endtime,
        skip=self.skip,
        frames=self.frames,
        rotate=self.rotate,
        contrast=self.contrast,
        brightness=self.brightness,
        crop=self.crop,
        color=self.color,
        returned_by_process=self.returned_by_process,
        keep_all=self.keep_all,
    )

    # Convert eventual low-resolution video or image
    video_formats = [
        ".avi",
        ".mp4",
        ".mov",
        ".mkv",
        ".mpg",
        ".mpeg",
        ".webm",
        ".ogg",
        ".ts",
        ".wmv",
        ".3gp",
        ".360",
    ]
    if self.fex not in video_formats:
        # Check if it is an image file
        if get_framecount(self.filename) == 1:
            image_formats = [
                ".gif",
                ".jpeg",
                ".jpg",
                ".jfif",
                ".pjpeg",
                ".png",
                ".svg",
                ".webp",
                ".avif",
                ".apng",
            ]
            if self.fex not in image_formats:
                # Create one converted version and register it to the MgVideo
                filename = convert(
                    self.of + self.fex, self.of + self.fex + ".png", overwrite=True
                )
                # point of and fex to the png version
                self.of, self.fex = os.path.splitext(filename)
            else:
                # update filename after the processes
                self.filename = self.of + self.fex
        else:
            # Create one converted version and register it to the MgVideo
            filename = convert_to_mp4(self.of + self.fex, overwrite=True)
            # point of and fex to the mp4 version
            self.of, self.fex = os.path.splitext(filename)
    else:
        # Update filename after the processes
        self.filename = self.of + self.fex

    # Check if there is audio in the video file
    if self.has_audio:
        self.audio = MgAudio(self.filename, self.sr, self.n_fft, self.hop_length)
    else:
        self.audio = None

numpy

numpy()

Read all video frames into a numpy array using FFmpeg.

Returns:

Name	Type	Description
tuple		A tuple (array, fps) where array is a numpy.ndarray of shape (N, H, W, 3) in BGR format (uint8) containing all N frames, and fps is the frame rate of the video.

Source code in musicalgestures/_video.py

def numpy(self):
    """
    Read all video frames into a numpy array using FFmpeg.

    Returns:
        tuple: A tuple ``(array, fps)`` where ``array`` is a ``numpy.ndarray``
            of shape ``(N, H, W, 3)`` in BGR format (uint8) containing all N
            frames, and ``fps`` is the frame rate of the video.
    """
    # Define ffmpeg command and load all the video frames in memory
    cmd = ["ffmpeg", "-y", "-i", self.filename]
    process = ffmpeg_cmd(cmd, total_time=self.length, pipe="load")
    # Convert bytes to numpy array
    array = np.frombuffer(process.stdout, dtype=np.uint8).reshape(
        -1, self.height, self.width, 3
    )

    return array, self.fps

from_numpy

from_numpy(array, fps, target_name=None)

Writes a numpy array of video frames to a video file using FFmpeg.

After writing, updates self.filename, self.of, and self.fex to reflect the actual output path so that subsequent operations on this object refer to the newly created file.

Parameters:

Name	Type	Description	Default
array	ndarray	Video frames array with shape (N, H, W, 3) in BGR format.	required
fps	float	Frames per second for the output video.	required
target_name	str	Full path for the output file. If None, uses self.path/self.filename (or just self.filename if path is None). Defaults to None.	None

Source code in musicalgestures/_video.py

def from_numpy(self, array: np.ndarray, fps: float, target_name: str | None = None) -> None:
    """
    Writes a numpy array of video frames to a video file using FFmpeg.

    After writing, updates ``self.filename``, ``self.of``, and ``self.fex`` to
    reflect the actual output path so that subsequent operations on this object
    refer to the newly created file.

    Args:
        array (np.ndarray): Video frames array with shape (N, H, W, 3) in BGR format.
        fps (float): Frames per second for the output video.
        target_name (str, optional): Full path for the output file. If None, uses
            ``self.path/self.filename`` (or just ``self.filename`` if path is None).
            Defaults to None.
    """
    if target_name is not None:
        write_path = os.path.splitext(target_name)[0] + self.fex
    elif self.path is not None:
        write_path = os.path.join(self.path, self.filename)
    else:
        write_path = self.filename

    process = None
    for frame in array:
        if process is None:
            cmd = [
                "ffmpeg",
                "-y",
                "-s",
                "{}x{}".format(frame.shape[1], frame.shape[0]),
                "-r",
                str(fps),
                "-f",
                "rawvideo",
                "-pix_fmt",
                "bgr24",
                "-vcodec",
                "rawvideo",
                "-i",
                "-",
                "-vcodec",
                "libx264",
                "-pix_fmt",
                "yuv420p",
                write_path,
            ]
            process = ffmpeg_cmd(cmd, total_time=array.shape[0], pipe="write")
        process.stdin.write(frame.astype(np.uint8))
    # the loop opens the process on its first frame, so this only holds for a non-empty array
    assert process is not None, "no frames were written: the array is empty"
    process.stdin.close()
    process.wait()

    # Update self.filename to the actual written path so that get_video() can find the file
    self.filename = write_path
    self.of, self.fex = os.path.splitext(write_path)

extract_frame

extract_frame(**kwargs)

Extracts a frame from the video at a given time. see _utils.extract_frame for details.

Other Parameters:

Name	Type	Description
frame	int	The frame number to extract.
time	str	The time in HH:MM:ss.ms where to extract the frame from.
target_name	str	The name for the output file. If None, the name will be FRAME.. Defaults to None.
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filename to avoid overwriting. Defaults to True.

Returns:

Name	Type	Description
MgImage		An MgImage object referring to the extracted frame.

Source code in musicalgestures/_video.py

def extract_frame(self, **kwargs):
    """
    Extracts a frame from the video at a given time.
    see _utils.extract_frame for details.

    Keyword Args:
        frame (int): The frame number to extract.
        time (str): The time in HH:MM:ss.ms where to extract the frame from.
        target_name (str, optional): The name for the output file. If None, the name will be <input name>FRAME<frame number>.<file extension>. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filename to avoid overwriting. Defaults to True.

    Returns:
        MgImage: An MgImage object referring to the extracted frame.
    """
    return MgImage(extract_frame(self.filename, **kwargs))