Video

Bases: MgAudio

This is the class for working with video files in the Musical Gestures Toolbox. It inherites from the class MgAudio for working with audio files as well. There is a set of preprocessing tools you can use when you load a video, such as: - trimming: to extract a section of the video, - skipping: to shrink the video by skipping N frames after keeping one, - rotating: to rotate the video by N degrees, - applying brightness and contrast - cropping: to crop the video either automatically (by assessing the area of motion) or manually with a pop-up user interface, - converting to grayscale

These preprocesses will apply upon creating the MgVideo. Further processes are available as class methods.

Initializes Musical Gestures data structure from a video file, and applies preprocesses if desired.

Parameters:

Name	Type	Description	Default
filename	Union[str, List[str]]	Path to the video file. If input is a list, will merge all videos into one.	required
array	ndarray	Generates an MgVideo object from a video array. Defaults to None.	None
fps	float	The frequency at which consecutive images from the video array are captured or displayed. Defaults to None.	None
path	str	Path to save the output video file generated from a video array. Defaults to None.	None
filtertype	str	The filtertype parameter for the motion() method. Regular turns all values below threshold to 0. Binary turns all values below threshold to 0, above threshold to 1. Blob removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	The threshold parameter for the motion() method. Eliminates pixel values less than given threshold. A number in the range of 0 to 1. Defaults to 0.05.	0.05
starttime	int or float	Trims the video from this start time (s). Defaults to 0.	0
endtime	int or float	Trims the video until this end time (s). Defaults to 0 (which means the full length).	0
blur	str	The blur parameter for the motion() method. 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
skip	int	Time-shrinks the video by skipping (discarding) every n frames determined by skip. Defaults to 0.	0
frames	int	Specify a fixed target number of frames to extract from the video. Defaults to 0.	0
rotate	int	Rotates the video by a rotate degrees. Defaults to 0.	0
color	bool	If False, converts the video to grayscale and sets every method in grayscale mode. Defaults to True.	True
contrast	int	Applies +/- 100 contrast to video. Defaults to 0.	0
brightness	int	Applies +/- 100 brightness to video. Defaults to 0.	0
crop	str	If 'manual', opens a window displaying the first frame of the input video file, where the user can draw a rectangle to which cropping is applied. If 'auto' the cropping function attempts to determine the area of significant motion and applies the cropping to that area. Defaults to 'None'.	'None'
keep_all	bool	If True, preserves an output video file after each used preprocessing stage. Defaults to False.	False
returned_by_process	bool	This parameter is only for internal use, do not use it. Defaults to False.	False
sr	int	Sampling rate of the audio file. Defaults to 22050.	22050
n_fft	int	Length of the FFT window. Defaults to 2048.	2048
hop_length	int	Number of samples between successive frames. Defaults to 512.	512

Source code in musicalgestures/_video.py

def __init__(
    self,
    filename: Union[str, List[str]],
    array=None,
    fps: float = None,
    path: str = None,
    # Video parameters
    filtertype: str = "Regular",
    threshold: float = 0.05,
    starttime: float = 0,
    endtime: float = 0,
    blur: str = "None",
    skip: int = 0,
    frames: int = 0,
    rotate: float = 0,
    color: bool = True,
    contrast: float = 0,
    brightness: float = 0,
    crop: str = "None",
    keep_all: bool = False,
    returned_by_process: bool = False,
    # Audio parameters
    sr: int = 22050,
    n_fft: int = 2048,
    hop_length: int = 512,
):
    """
    Initializes Musical Gestures data structure from a video file, and applies preprocesses if desired.

    Args:
        filename (Union[str, List[str]]): Path to the video file. If input is a list, will merge all videos into one.
        array (np.ndarray, optional): Generates an MgVideo object from a video array. Defaults to None.
        fps (float, optional): The frequency at which consecutive images from the video array are captured or displayed. Defaults to None.
        path (str, optional): Path to save the output video file generated from a video array. Defaults to None.
        filtertype (str, optional): The `filtertype` parameter for the `motion()` method. `Regular` turns all values below `threshold` to 0. `Binary` turns all values below `threshold` to 0, above `threshold` to 1. `Blob` removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): The `threshold` parameter for the `motion()` method. Eliminates pixel values less than given threshold. A number in the range of 0 to 1. Defaults to 0.05.
        starttime (int or float, optional): Trims the video from this start time (s). Defaults to 0.
        endtime (int or float, optional): Trims the video until this end time (s). Defaults to 0 (which means the full length).
        blur (str, optional): The `blur` parameter for the `motion()` method. 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        skip (int, optional): Time-shrinks the video by skipping (discarding) every n frames determined by `skip`. Defaults to 0.
        frames (int, optional): Specify a fixed target number of frames to extract from the video. Defaults to 0.
        rotate (int, optional): Rotates the video by a `rotate` degrees. Defaults to 0.
        color (bool, optional): If False, converts the video to grayscale and sets every method in grayscale mode. Defaults to True.
        contrast (int, optional): Applies +/- 100 contrast to video. Defaults to 0.
        brightness (int, optional): Applies +/- 100 brightness to video. Defaults to 0.
        crop (str, optional): If 'manual', opens a window displaying the first frame of the input video file, where the user can draw a rectangle to which cropping is applied. If 'auto' the cropping function attempts to determine the area of significant motion and applies the cropping to that area. Defaults to 'None'.
        keep_all (bool, optional): If True, preserves an output video file after each used preprocessing stage. Defaults to False.
        returned_by_process (bool, optional): This parameter is only for internal use, do not use it. Defaults to False.

        sr (int, optional): Sampling rate of the audio file. Defaults to 22050.
        n_fft (int, optional): Length of the FFT window. Defaults to 2048.
        hop_length (int, optional): Number of samples between successive frames. Defaults to 512.
    """

    # if filename is a list, merge all videos into one
    if isinstance(filename, list):
        self.filename = merge_videos(filename)
    else:
        self.filename = filename

    self.array = array
    self.fps = fps
    self.path = path
    # Name of file without extension (only-filename)
    self.of = os.path.splitext(self.filename)[0]
    self.fex = os.path.splitext(self.filename)[1]
    # Video parameters
    self.color = color
    self.starttime = starttime
    self.endtime = endtime
    self.skip = skip
    self.frames = frames
    self.filtertype = filtertype
    self.threshold = threshold
    self.blur = blur
    self.contrast = contrast
    self.brightness = brightness
    self.crop = crop
    self.rotate = rotate
    self.keep_all = keep_all
    self.has_audio = None
    self.returned_by_process = returned_by_process
    # Audio parameters
    self.sr = sr
    self.n_fft = n_fft
    self.hop_length = hop_length

    # Check input and if FFmpeg is properly installed
    self.test_input()

    if all(arg is not None for arg in [self.array, self.fps]):
        self.from_numpy(self.array, self.fps)

    self.get_video()
    self.flow = Flow(self, self.filename, self.color, self.has_audio)

filename instance-attribute

filename = merge_videos(filename)

array instance-attribute

array = array

fps instance-attribute

fps = fps

path instance-attribute

path = path

of instance-attribute

of = os.path.splitext(self.filename)[0]

fex instance-attribute

fex = os.path.splitext(self.filename)[1]

color instance-attribute

color = color

starttime instance-attribute

starttime = starttime

endtime instance-attribute

endtime = endtime

skip instance-attribute

skip = skip

frames instance-attribute

frames = frames

filtertype instance-attribute

filtertype = filtertype

threshold instance-attribute

threshold = threshold

blur instance-attribute

blur = blur

contrast instance-attribute

contrast = contrast

brightness instance-attribute

brightness = brightness

crop instance-attribute

crop = crop

rotate instance-attribute

rotate = rotate

keep_all instance-attribute

keep_all = keep_all

has_audio instance-attribute

has_audio = None

returned_by_process instance-attribute

returned_by_process = returned_by_process

sr instance-attribute

sr = sr

n_fft instance-attribute

n_fft = n_fft

hop_length instance-attribute

hop_length = hop_length

flow instance-attribute

flow = Flow(self, self.filename, self.color, self.has_audio)

n_frames property

n_frames

Number of frames in the video (an alias for the frame-count length).

duration property

duration

Video duration in seconds (length / fps).

Note self.length is the frame count for an MgVideo (it is the duration in seconds for an MgAudio); use this property when you want seconds.

motion

motion(filtertype='Regular', threshold=0.05, blur='None', kernel_size=5, use_median=False, unit='seconds', atadenoise=False, motion_analysis='all', inverted_motionvideo=False, inverted_motiongram=False, equalize_motiongram=False, audio_descriptors=False, save_plot=True, title=None, save_data=True, data_format='csv', save_motiongrams=True, save_video=True, target_name_video=None, target_name_plot=None, target_name_data=None, target_name_mgx=None, target_name_mgy=None, overwrite=True, normalize=None)

Finds the difference in pixel value from one frame to the next in an input video, and saves the frames into a new video. Describes the motion in the recording. Outputs: a motion video, a plot describing the centroid of motion and the quantity of motion, horizontal and vertical motiongrams, and a text file containing the quantity of motion and the centroid of motion for each frame with timecodes in milliseconds.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
kernel_size	int	Size of structuring element. Defaults to 5.	5
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
unit	str	Unit in QoM plot. Accepted values are 'seconds' or 'samples'. Defaults to 'seconds'.	'seconds'
atadenoise	bool	If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.	False
normalize	bool | None	Chooses between data for looking at and data for measuring with. True (and None, the long-standing default) writes a Qom column divided by that clip's own maximum, so every clip peaks at 1.0 -- what a plot on a 0-1 axis wants, and the reason the value exists. It also makes quantity of motion incomparable between clips and lets one bright frame set the scale. False writes QomRaw, the untouched sum of pixel values in the thresholded difference frame: comparable across clips, on the scale the pixels had. The column is renamed rather than rescaled so a file cannot be misread later.	None
motion_analysis	str	Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.	'all'
inverted_motionvideo	bool	If True, inverts colors of the motion video. Defaults to False.	False
inverted_motiongram	bool	If True, inverts colors of the motiongrams. Defaults to False.	False
equalize_motiongram	bool	If True, converts the motiongrams to hsv-color space and flattens the value channel (v). Defaults to True.	False
save_plot	bool	If True, outputs motion-plot. Defaults to True.	True
title	str	Optionally add title to the plot. Defaults to None, which uses the file name as a title.	None
save_data	bool	If True, outputs motion-data. Defaults to True.	True
data_format	str / list	Specifies format of motion-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.	'csv'
save_motiongrams	bool	If True, outputs motiongrams. Defaults to True.	True
save_video	bool	If True, outputs the motion video. Defaults to True.	True
target_name_video	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).	None
target_name_plot	str	Target output name for the plot. Defaults to None (which assumes that the input filename with the suffix "_motion_com_aom_qom" should be used).	None
target_name_data	str	Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).	None
target_name_mgx	str	Target output name for the vertical motiongram. Defaults to None (which assumes that the input filename with the suffix "_mgv" should be used).	None
target_name_mgy	str	Target output name for the horizontal motiongram. Defaults to None (which assumes that the input filename with the suffix "_mgh" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	'musicalgestures.MgVideo'	A new MgVideo pointing to the output video file. If save_video=False, it returns an MgVideo pointing to the input video file.

Source code in musicalgestures/_motionvideo.py

def mg_motion(
        self,
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        kernel_size: int = 5,
        use_median: bool = False,
        unit: str = 'seconds',
        atadenoise: bool = False,
        motion_analysis: str = 'all',
        inverted_motionvideo: bool = False,
        inverted_motiongram: bool = False,
        equalize_motiongram: bool = False,
        audio_descriptors: bool = False,
        save_plot: bool = True,
        title: str | None = None,
        save_data: bool = True,
        data_format: str | list = "csv",
        save_motiongrams: bool = True,
        save_video: bool = True,
        target_name_video: str | None = None,
        target_name_plot: str | None = None,
        target_name_data: str | None = None,
        target_name_mgx: str | None = None,
        target_name_mgy: str | None = None,
        overwrite: bool = True,
        normalize: bool | None = None) -> "musicalgestures.MgVideo":
    """
    Finds the difference in pixel value from one frame to the next in an input video, and saves the frames into a new video. 
    Describes the motion in the recording. Outputs: a motion video, a plot describing the centroid of motion and the 
    quantity of motion, horizontal and vertical motiongrams, and a text file containing the quantity of motion and the 
    centroid of motion for each frame with timecodes in milliseconds.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        unit (str, optional): Unit in QoM plot. Accepted values are 'seconds' or 'samples'. Defaults to 'seconds'.
        atadenoise (bool, optional): If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.
        normalize (bool | None, optional): Chooses between data for looking at and data for
            measuring with. True (and None, the long-standing default) writes a `Qom` column
            divided by that clip's own maximum, so every clip peaks at 1.0 -- what a plot on a
            0-1 axis wants, and the reason the value exists. It also makes quantity of motion
            **incomparable between clips** and lets one bright frame set the scale. False writes
            `QomRaw`, the untouched sum of pixel values in the thresholded difference frame:
            comparable across clips, on the scale the pixels had. The column is renamed rather
            than rescaled so a file cannot be misread later.
        motion_analysis (str, optional): Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.
        inverted_motionvideo (bool, optional): If True, inverts colors of the motion video. Defaults to False.
        inverted_motiongram (bool, optional): If True, inverts colors of the motiongrams. Defaults to False.
        equalize_motiongram (bool, optional): If True, converts the motiongrams to hsv-color space and flattens the value channel (v). Defaults to True.
        save_plot (bool, optional): If True, outputs motion-plot. Defaults to True.
        title (str, optional): Optionally add title to the plot. Defaults to None, which uses the file name as a title.
        save_data (bool, optional): If True, outputs motion-data. Defaults to True.
        data_format (str/list, optional): Specifies format of motion-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.
        save_motiongrams (bool, optional): If True, outputs motiongrams. Defaults to True.
        save_video (bool, optional): If True, outputs the motion video. Defaults to True.
        target_name_video (str, optional): Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).
        target_name_plot (str, optional): Target output name for the plot. Defaults to None (which assumes that the input filename with the suffix "_motion_com_aom_qom" should be used).
        target_name_data (str, optional): Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).
        target_name_mgx (str, optional): Target output name for the vertical motiongram. Defaults to None (which assumes that the input filename with the suffix "_mgv" should be used).
        target_name_mgy (str, optional): Target output name for the horizontal motiongram. Defaults to None (which assumes that the input filename with the suffix "_mgh" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A new MgVideo pointing to the output video file. If `save_video=False`, it returns an MgVideo pointing to the input video file.
    """

    if save_plot | save_data | save_motiongrams | save_video:
        # ignore runtime warnings when dividing by 0
        np.seterr(divide='ignore', invalid='ignore')

        of, fex = self.of, self.fex

        # Define ffmpeg command start and end
        cmd = ['ffmpeg', '-y', '-i', self.filename]
        # Filter video frames using ffmpeg
        cmd, cmd_filter = filter_frame_ffmpeg(self.filename, cmd, self.color, blur, filtertype, threshold, kernel_size, use_median)

        if atadenoise:
            # Apply an adaptive temporal averaging denoiser every 129 frames
            cmd_filter += 'atadenoise=s=129'
        else:
            # Remove last comma after previous filter
            cmd_filter = cmd_filter[: -1]
        cmd += ['-filter_complex', cmd_filter] 

        if save_motiongrams:
            gramx = np.zeros([1, self.width, 3]).astype(np.uint8)
            gramy = np.zeros([self.height, 1, 3]).astype(np.uint8) 

        if save_data | save_plot:      
            time = np.array([]) # time in ms
            aom = np.array([])  # area of motion
            qom = np.array([])  # quantity of motion
            com = np.array([])  # centroid of motion

        if save_video:
            if target_name_video is None:
                target_name_video = of + '_motion' + fex
            # enforce avi
            else:
                target_name_video = os.path.splitext(target_name_video)[0] + fex
            if not overwrite:
                target_name_video = generate_outfilename(target_name_video)

        pgbar_text = 'Rendering motion' + ", ".join(np.array(["-video", "-grams", "-plots", "-data"])[
            np.array([save_video, save_motiongrams, save_plot, save_data])]) + ":" 
        pb = MgProgressbar(total=self.length, prefix=pgbar_text)  

        # Pipe video with FFmpeg for reading frame by frame        
        process = ffmpeg_cmd(cmd, total_time=self.length, pipe='read')
        video_out = None

        i = 0
        while True:
            # Read frame-by-frame
            out = process.stdout.read(self.width*self.height*3)

            if out == b'':
                pb.progress(self.length)
                break

            # Transform the bytes read into a numpy array
            motion_frame = np.frombuffer(out, dtype=np.uint8).reshape([self.height, self.width, 3]) # height, width, channels

            if save_data | save_plot:
                if motion_analysis.lower() == 'aom':
                    # Area of Motion (AoM)
                    aombite = area(motion_frame, self.height, self.width)
                    if i == 0:
                        time = frame2ms(i, self.fps)
                        aom = np.array(aombite).reshape(1, 4)
                    else:
                        time = np.append(time, frame2ms(i, self.fps))
                        aom = np.append(aom, np.array(aombite).reshape(1, 4), axis=0)

                if motion_analysis.lower() == 'com' or motion_analysis.lower() == 'qom':
                    # Centroid of Motion (CoM) and Quantity of Motion (QoM)
                    combite, qombite = centroid(motion_frame, self.width, self.height)
                    if i == 0:
                        time = frame2ms(i, self.fps)
                        com = combite.reshape(1, 2)
                        qom = qombite
                    else:
                        time = np.append(time, frame2ms(i, self.fps))
                        com = np.append(com, combite.reshape(1, 2), axis=0)
                        qom = np.append(qom, qombite)

                if motion_analysis.lower() == 'all':
                    # Area of Motion (AoM)
                    aombite = area(motion_frame, self.height, self.width)                    
                    # Centroid of Motion (CoM) and Quantity of Motion (QoM)
                    combite, qombite = centroid(motion_frame, self.width, self.height)
                    if i == 0:
                        time = frame2ms(i, self.fps)
                        com = combite.reshape(1, 2)
                        qom = qombite
                        aom = np.array(aombite).reshape(1, 4)
                    else:
                        time = np.append(time, frame2ms(i, self.fps))
                        com = np.append(com, combite.reshape(1, 2), axis=0)
                        qom = np.append(qom, qombite)
                        aom = np.append(aom, np.array(aombite).reshape(1, 4), axis=0)

            if save_motiongrams:
                movement_y = np.mean(motion_frame, axis=1).reshape(self.height, 1, 3).astype(np.uint8)
                movement_x = np.mean(motion_frame, axis=0).reshape(1, self.width, 3).astype(np.uint8)

                gramy = np.append(gramy, movement_y, axis=1).astype(np.uint8)
                gramx = np.append(gramx, movement_x, axis=0).astype(np.uint8)

            if save_video:
                if video_out is None:
                    cmd =['ffmpeg', '-y', '-s', '{}x{}'.format(motion_frame.shape[1], motion_frame.shape[0]), 
                        '-r', str(self.fps), '-f', 'rawvideo', '-pix_fmt', 'bgr24', '-vcodec', 'rawvideo', 
                        '-i', '-', '-vcodec', 'libx264', '-pix_fmt', 'yuv420p', target_name_video]
                    video_out = ffmpeg_cmd(cmd, total_time=self.length, pipe='write')

                if inverted_motionvideo:
                    video_out.stdin.write(cv2.bitwise_not(motion_frame.astype(np.uint8)))
                else:
                    video_out.stdin.write(motion_frame.astype(np.uint8))

            # Flush the buffer
            process.stdout.flush()
            pb.progress(i)
            i += 1

        # Terminate the processes
        if save_video:
            video_out.stdin.close()
            video_out.wait()
        process.terminate()

        if save_motiongrams:
            gramx = (gramx-gramx.min())/(gramx.max()-gramx.min())*255.0
            gramy = (gramy-gramy.min())/(gramy.max()-gramy.min())*255.0

            if equalize_motiongram:
                gramx = gramx.astype(np.uint8)
                gramx_hsv = cv2.cvtColor(gramx, cv2.COLOR_RGB2HSV).astype(np.uint8)
                gramx_hsv[:, :, 2] = cv2.equalizeHist(gramx_hsv[:, :, 2]).astype(np.uint8)
                gramx = cv2.cvtColor(gramx_hsv, cv2.COLOR_HSV2RGB).astype(np.uint8)

                gramy = gramy.astype(np.uint8)
                gramy_hsv = cv2.cvtColor(gramy, cv2.COLOR_RGB2HSV).astype(np.uint8)
                gramy_hsv[:, :, 2] = cv2.equalizeHist(gramy_hsv[:, :, 2]).astype(np.uint8)
                gramy = cv2.cvtColor(gramy_hsv, cv2.COLOR_HSV2RGB).astype(np.uint8)

            if target_name_mgx is None:
                target_name_mgx = of + '_mgv.png'
            if target_name_mgy is None:
                target_name_mgy = of + '_mgh.png'
            if not overwrite:
                target_name_mgx = generate_outfilename(target_name_mgx)
                target_name_mgy = generate_outfilename(target_name_mgy)

            if inverted_motiongram:
                cv2.imwrite(target_name_mgx, cv2.bitwise_not(gramx.astype(np.uint8)))
                cv2.imwrite(target_name_mgy, cv2.bitwise_not(gramy.astype(np.uint8)))
            else:
                cv2.imwrite(target_name_mgx, gramx.astype(np.uint8))
                cv2.imwrite(target_name_mgy, gramy.astype(np.uint8))

            # save motiongrams data and convert to grayscale for processing motiongrams Self-Similarity Matrices (SSMs)
            data = (cv2.cvtColor(gramx.astype(np.uint8), cv2.COLOR_RGB2GRAY), cv2.cvtColor(gramy.astype(np.uint8), cv2.COLOR_RGB2GRAY))
            self.ssm_fig = MgFigure(figure=None, figure_type='video.ssm', data=data, layers=None, image=(target_name_mgx, target_name_mgy))

            # save rendered motiongrams as MgImages into parent MgVideo
            self.motiongram_x = MgImage(target_name_mgx)
            self.motiongram_y = MgImage(target_name_mgy)

        if audio_descriptors:
            audio_descriptors = self

        if save_data:
            # `normalize` was declared on mg_motion and never used. It now
            # chooses between the per-clip normalised `Qom` column, which is
            # what a plot wants, and the raw `QomRaw` sum of pixel values,
            # which is what a cross-clip measurement needs. None keeps the
            # long-standing default so no existing call changes.
            save_txt(of, time, aom, com, qom, motion_analysis, self.width, self.height, 
            data_format=data_format, target_name_data=target_name_data, overwrite=overwrite,
            normalize=True if normalize is None else bool(normalize))

        if save_plot:
            if title is None:
                title = os.path.basename(of + fex)
            # save plot as an MgImage at motion_plot for parent MgVideo
            self.motion_plot = MgImage(save_analysis(of, self.fps, aom, com, qom, motion_analysis, audio_descriptors, self.width,
                                        self.height, unit, title, target_name_plot=target_name_plot, overwrite=overwrite))

        # Resetting numpy warnings for dividing by 0
        np.seterr(divide='warn', invalid='warn')

        if save_video:
            # Check if the original video fil has audio
            if self.has_audio:
                source_audio = extract_wav(of + fex)
                embed_audio_in_video(source_audio, target_name_video)
                os.remove(source_audio)

            # Save generated musicalgestures video as the video of the parent MgVideo
            self.motion_video = musicalgestures.MgVideo(filename=target_name_video, returned_by_process=True)
            return self.motion_video

        else:
            return self

    else:
        # Return the MgVideo the motion() was called upon
        print("Nothing to render. Exiting...")
        return self

motiongrams

motiongrams(filtertype='Regular', threshold=0.05, blur='None', use_median=False, atadenoise=False, kernel_size=5, inverted_motiongram=False, equalize_motiongram=True, target_name_mgx=None, target_name_mgy=None, overwrite=True, normalize=None)

Shortcut for mg_motion to only render motiongrams.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
atadenoise	bool	If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.	False
kernel_size	int	Size of the median filter (if use_median=True) or the erosion filter (if filtertype='blob'). Defaults to 5.	5
inverted_motiongram	bool	If True, inverts colors of the motiongrams. Defaults to False.	False
equalize_motiongram	bool	If True, converts the motiongrams to hsv-color space and flattens the value channel (v). Defaults to True.	True
target_name_mgx	str	Target output name for the vertical motiongram. Defaults to None (which assumes that the input filename with the suffix "_mgv" should be used).	None
target_name_mgy	str	Target output name for the horizontal motiongram. Defaults to None (which assumes that the input filename with the suffix "_mgh" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgList	'MgList'	An MgList pointing to the output motiongram images (as MgImages).

Source code in musicalgestures/_motionvideo.py

def mg_motiongrams(
        self,
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        use_median: bool = False,
        atadenoise: bool = False,
        kernel_size: int = 5,
        inverted_motiongram: bool = False,
        equalize_motiongram: bool = True,
        target_name_mgx: str | None = None,
        target_name_mgy: str | None = None,
        overwrite: bool = True,
        normalize: bool | None = None) -> "MgList":
    """
    Shortcut for `mg_motion` to only render motiongrams.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        atadenoise (bool, optional): If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.
        kernel_size (int, optional): Size of the median filter (if `use_median=True`) or the erosion filter (if `filtertype='blob'`). Defaults to 5.
        inverted_motiongram (bool, optional): If True, inverts colors of the motiongrams. Defaults to False.
        equalize_motiongram (bool, optional): If True, converts the motiongrams to hsv-color space and flattens the value channel (v). Defaults to True.
        target_name_mgx (str, optional): Target output name for the vertical motiongram. Defaults to None (which assumes that the input filename with the suffix "_mgv" should be used).
        target_name_mgy (str, optional): Target output name for the horizontal motiongram. Defaults to None (which assumes that the input filename with the suffix "_mgh" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgList: An MgList pointing to the output motiongram images (as MgImages).
    """

    if target_name_mgx is None:
        target_name_mgx = self.of + '_mgv.png'
    if target_name_mgy is None:
        target_name_mgy = self.of + '_mgh.png'
    if not overwrite:
        out_x = generate_outfilename(target_name_mgx)
        out_y = generate_outfilename(target_name_mgy)
    else:
        out_x, out_y = target_name_mgx, target_name_mgy

    mg_motion(
        self,
        filtertype=filtertype,
        threshold=threshold,
        blur=blur,
        kernel_size=kernel_size,
        use_median=use_median,
        atadenoise=atadenoise,
        inverted_motiongram=inverted_motiongram,
        equalize_motiongram=equalize_motiongram,
        save_data=False,
        save_motiongrams=True,
        save_plot=False,
        save_video=False,
        target_name_mgx=out_x,
        target_name_mgy=out_y,
        overwrite=True)

    # mg_motion also saves the motiongrams as MgImages to self.motiongram_x and self.motiongram_y of the parent MgVideo
    return MgList(MgImage(out_x), MgImage(out_y))

motiondata

motiondata(filtertype='Regular', threshold=0.05, blur='None', kernel_size=5, atadenoise=False, use_median=False, motion_analysis='all', data_format='csv', target_name=None, overwrite=True)

Shortcut for mg_motion to only render motion data.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
kernel_size	int	Size of structuring element. Defaults to 5.	5
atadenoise	bool	If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.	False
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
motion_analysis	str	Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.	'all'
data_format	str / list	Specifies format of motion-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.	'csv'
target_name	str	Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Type	Description
'list'	str/list: The path(s) to the rendered data file(s).

Source code in musicalgestures/_motionvideo.py

def mg_motiondata(
        self,
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        kernel_size: int = 5,
        atadenoise: bool = False,
        use_median: bool = False,
        motion_analysis: str = 'all',
        data_format: str | list = "csv",
        target_name: str | None = None,
        overwrite: bool = True) -> "list":
    """
    Shortcut for `mg_motion` to only render motion data.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        atadenoise (bool, optional): If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        motion_analysis (str, optional): Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.
        data_format (str/list, optional): Specifies format of motion-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, eg. ['csv', 'txt']. Defaults to 'csv'.
        target_name (str, optional): Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        str/list: The path(s) to the rendered data file(s).
    """

    out = None

    if type(data_format) == str:
        if target_name is None:
            target_name = self.of + '_motion.' + data_format
        if not overwrite:
            target_name = generate_outfilename(target_name)
        out = target_name

    if type(data_format) == list:
        out = []
        if target_name is None:
            # this csv is just a temporary placeholder, the correct extension is always enforced based on the data_format(s)
            target_name = self.of + '_motion.csv'
        target_name_of = os.path.splitext(target_name)[0]
        for item in data_format:
            if not overwrite:
                tmp_name = generate_outfilename(target_name_of + '.' + item)
            else:
                tmp_name = target_name_of + '.' + item
            out.append(tmp_name)

    mg_motion(
        self,
        filtertype=filtertype,
        threshold=threshold,
        blur=blur,
        kernel_size=kernel_size,
        use_median=use_median,
        atadenoise=atadenoise,
        motion_analysis=motion_analysis,
        data_format=data_format,
        save_data=True,
        save_motiongrams=False,
        save_plot=False,
        save_video=False,
        target_name_data=target_name,
        overwrite=overwrite)

    # if type(data_format) == list:
    #     outlist = [self.of + '_motion.' + elem for elem in data_format]
    #     return outlist
    # else:
    #     return self.of + '_motion.' + data_format
    return out

motionplots

motionplots(filtertype='Regular', threshold=0.05, blur='None', kernel_size=5, use_median=False, atadenoise=False, motion_analysis='all', audio_descriptors=False, unit='seconds', title=None, target_name=None, overwrite=True)

Shortcut for mg_motion to only render motion plots.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
kernel_size	int	Size of structuring element. Defaults to 5.	5
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
atadenoise	bool	If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.	False
motion_analysis	str	Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.	'all'
audio_descriptors	bool	Whether to plot motion plots together with audio descriptors in order to see possible correlations in the data. Defaults to False.	False
unit	str	Unit in QoM plot. Accepted values are 'seconds' or 'samples'. Defaults to 'seconds'.	'seconds'
title	str	Optionally add title to the plot. Defaults to None, which uses the file name as a title.	None
target_name	str	Target output name for the plot. Defaults to None (which assumes that the input filename with the suffix "_motion_com_aom_qom" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	An MgImage pointing to the exported image (png) of the motion plots.

Source code in musicalgestures/_motionvideo.py

def mg_motionplots(
        self,
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        kernel_size: int = 5,
        use_median: bool = False,
        atadenoise: bool = False,
        motion_analysis: str = 'all',
        audio_descriptors: bool = False,
        unit: str = 'seconds',
        title: str | None = None,
        target_name: str | None = None,
        overwrite: bool = True) -> "MgImage":
    """
    Shortcut for `mg_motion` to only render motion plots.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        atadenoise (bool, optional): If True, applies an adaptive temporal averaging denoiser every 129 frames. Defaults to False.
        motion_analysis (str, optional): Specify which motion analysis to process or all. 'AoM' renders the Area of Motion. 'CoM' renders the Centroid of Motion. 'QoM' renders the Quantity of Motion. 'all' renders all the motion analysis available. Defaults to 'all'.
        audio_descriptors (bool, optional): Whether to plot motion plots together with audio descriptors in order to see possible correlations in the data. Defaults to False.
        unit (str, optional): Unit in QoM plot. Accepted values are 'seconds' or 'samples'. Defaults to 'seconds'.
        title (str, optional): Optionally add title to the plot. Defaults to None, which uses the file name as a title.
        target_name (str, optional): Target output name for the plot. Defaults to None (which assumes that the input filename with the suffix "_motion_com_aom_qom" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgImage: An MgImage pointing to the exported image (png) of the motion plots.
    """

    if target_name is None:
        target_name = self.of + '_motionplots.png'
    if not overwrite:
        target_name = generate_outfilename(target_name)

    mg_motion(
        self,
        filtertype=filtertype,
        threshold=threshold,
        blur=blur,
        kernel_size=kernel_size,
        use_median=use_median,
        unit=unit,
        atadenoise=atadenoise,
        motion_analysis=motion_analysis,
        audio_descriptors=audio_descriptors,
        save_data=False,
        save_motiongrams=False,
        save_plot=True,
        title=title,
        save_video=False,
        target_name_plot=target_name,
        overwrite=overwrite)

    # mg_motion also saves the plot as an MgImage to self.motion_plot of the parent MgVideo
    return MgImage(target_name)

motionvideo

motionvideo(filtertype='Regular', threshold=0.05, blur='None', use_median=False, kernel_size=5, inverted_motionvideo=False, target_name=None, overwrite=True)

Shortcut to only render the motion video. Uses musicalgestures._utils.motionvideo_ffmpeg. Note that this does not apply median filter by default. If you need it use use_median=True.

Parameters:

Name	Type	Description	Default
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
kernel_size	int	Size of the median filter (if use_median=True) or the erosion filter (if filtertype='blob'). Defaults to 5.	5
inverted_motionvideo	bool	If True, inverts colors of the motion video. Defaults to False.	False
target_name	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	'musicalgestures.MgVideo'	A new MgVideo pointing to the output '_motion' video file.

Source code in musicalgestures/_motionvideo.py

def mg_motionvideo(
        self,
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        use_median: bool = False,
        kernel_size: int = 5,
        inverted_motionvideo: bool = False,
        target_name: str | None = None,
        overwrite: bool = True) -> "musicalgestures.MgVideo":
    """
    Shortcut to only render the motion video. Uses musicalgestures._utils.motionvideo_ffmpeg. Note that this does not apply median filter by default. If you need it use `use_median=True`.

    Args:
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        kernel_size (int, optional): Size of the median filter (if `use_median=True`) or the erosion filter (if `filtertype='blob'`). Defaults to 5.
        inverted_motionvideo (bool, optional): If True, inverts colors of the motion video. Defaults to False.
        target_name (str, optional): Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_motion" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A new MgVideo pointing to the output '_motion' video file.
    """

    motionvideo = motionvideo_ffmpeg(
        filename=self.filename,
        color=self.color,
        filtertype=filtertype,
        threshold=threshold,
        blur=blur,
        use_median=use_median,
        kernel_size=kernel_size,
        invert=inverted_motionvideo,
        target_name=target_name,
        overwrite=overwrite)

    # save motion video as motion_video for parent MgVideo
    # we have to do this here since we are not using mg_motion (that would normally save the result itself)
    self.motion_video = musicalgestures.MgVideo(motionvideo, color=self.color, returned_by_process=True)

    return self.motion_video

motionscore

motionscore()

Computes the average VMAF motion score of the video using FFmpeg.

Returns:

Name	Type	Description
float	'float'	The average VMAF motion score, or None if unavailable.

Source code in musicalgestures/_motionvideo.py

def mg_motionscore(self) -> "float":
    """
    Computes the average VMAF motion score of the video using FFmpeg.

    Returns:
        float: The average VMAF motion score, or None if unavailable.
    """
    cmd = ['ffmpeg', '-i', self.filename, '-vf', 'vmafmotion', '-f', 'null', '-']
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
    out, _ = process.communicate()
    splitted = out.split('\n')

    for index, item in enumerate(splitted):
        if isinstance(item, str) and re.search('VMAF', item):
            vmafmotion = float(re.findall(r'\d+\.\d+', splitted[index].split("] ")[1])[0])
            return vmafmotion
    print('VMAF motion score is not available.')
    return None

motion_mp

motion_mp(filtertype='Regular', threshold=0.05, blur='None', kernel_size=5, inverted_motionvideo=False, inverted_motiongram=False, unit='seconds', equalize_motiongram=True, save_plot=True, title=None, save_data=True, data_format='csv', save_motiongrams=True, save_video=True, target_name_video=None, target_name_plot=None, target_name_data=None, target_name_mgx=None, target_name_mgy=None, convert=True, overwrite=True, num_processes=-1)

Source code in musicalgestures/_motionvideo_mp_run.py

def mg_motion_mp(
        self,
        filtertype='Regular',
        threshold=0.05,
        blur='None',
        kernel_size=5,
        inverted_motionvideo=False,
        inverted_motiongram=False,
        unit='seconds',
        equalize_motiongram=True,
        save_plot=True,
        title=None,
        save_data=True,
        data_format="csv",
        save_motiongrams=True,
        save_video=True,
        target_name_video=None,
        target_name_plot=None,
        target_name_data=None,
        target_name_mgx=None,
        target_name_mgy=None,
        convert=True,
        overwrite=True,
        num_processes=-1) -> "musicalgestures.MgVideo":

    of, fex = self.of, self.fex

    # Convert to avi if the input is not avi - necesarry for cv2 compatibility on all platforms
    if convert and fex != '.avi':
        # first check if there already is a converted version, if not create one and register it to the parent self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex

    module_path = os.path.abspath(os.path.dirname(
        musicalgestures.__file__)).replace('\\', '/')
    the_system = platform.system()
    pythonkw = "python"
    if the_system != "Windows":
        pythonkw += "3"
    pyfile = wrap_str(module_path + '/_motionvideo_mp_render.py')

    temp_folder = tempfile.mkdtemp().replace('\\', '/')
    if temp_folder[-1] != "/":
        temp_folder += '/'
    # print("Temp folder:", temp_folder)

    of_feed = of.replace('\\', '/')
    of_feed = os.path.basename(of_feed)

    save_data_feed = save_data or save_plot

    command = [pythonkw, pyfile, temp_folder, of_feed, fex, self.fps, self.width, self.height, self.length, self.color, filtertype, threshold, blur,
               kernel_size, inverted_motionvideo, inverted_motiongram, equalize_motiongram, save_data_feed, save_motiongrams, save_video, num_processes]
    command = [str(item) for item in command]
    # print()
    # print(command)
    # print()

    pgbar_text = 'Rendering motion' + ", ".join(np.array(["-video", "-grams", "-plots", "-data"])[
        np.array([save_video, save_motiongrams, save_plot, save_data])]) + ":"
    pb = MgProgressbar(total=self.length, prefix=pgbar_text)
    progress = 0

    # set up socket server thread
    HOST = '127.0.0.1'  # Standard loopback interface address (localhost)
    PORT = 65432        # Port to listen on (non-privileged ports are > 1023)

    process = subprocess.Popen(command)

    num_cores = multiprocessing.cpu_count()
    # print("starting thread")
    t = threading.Thread(target=run_socket_server(HOST, PORT, pb, num_cores))
    t.start()

    try:
        # print("will wait")
        process.wait()

    except KeyboardInterrupt:
        try:
            process.terminate()
        except OSError:
            pass
        process.wait()
        # print("will join")
        t.join()
        # print("joined")
        raise KeyboardInterrupt

    # print("will join")
    t.join()
    # print("joined")

    # print("organizing results...")
    results = os.listdir(temp_folder)
    time_files = [temp_folder +
                  file for file in results if file.startswith("time")]
    time_files.sort()
    com_files = [temp_folder +
                 file for file in results if file.startswith("com")]
    com_files.sort()
    qom_files = [temp_folder +
                 file for file in results if file.startswith("qom")]
    qom_files.sort()
    gramx_files = [temp_folder +
                   file for file in results if file.startswith("gramx")]
    gramx_files.sort()
    gramy_files = [temp_folder +
                   file for file in results if file.startswith("gramy")]
    gramy_files.sort()
    video_files = [temp_folder +
                   file for file in results if file.endswith("avi")]
    video_files.sort()

    gramx, gramy, time, com, qom = None, None, None, None, None

    if save_motiongrams:
        # load gramx
        # if we only used a single chunk, load everything
        if len(gramx_files) == 1:
            gramx = np.load(gramx_files[0])
        # or in case there were multiple chunks...
        else:
            for idx, item in enumerate(gramx_files):
                if idx == 0:
                    # do not drop first row in first chunk
                    gramx = np.load(item)[:-1]
                elif idx == len(gramy_files) - 1:
                    # do not drop the last row in last chunk
                    gramx = np.append(gramx, np.load(item)[1:], axis=0)
                else:
                    # else drop first and last rows from chunk
                    gramx = np.append(gramx, np.load(item)[1:-1], axis=0)

        # load gramy
        # if we only used a single chunk, load everything
        if len(gramy_files) == 1:
            gramy = np.load(gramy_files[0])
        # or in case there were multiple chunks...
        else:
            for idx, item in enumerate(gramy_files):
                if idx == 0:
                    # do not drop first column in first chunk
                    gramy = np.load(item)[:, :-1]
                elif idx == len(gramy_files) - 1:
                    # do not drop the last column in last chunk
                    gramy = np.append(gramy, np.load(item)[:, 1:], axis=1)
                else:
                    # else drop first and last columns from chunk
                    gramy = np.append(gramy, np.load(item)[:, 1:-1], axis=1)

        if self.color == False:
            # Normalize before converting to uint8 to keep precision
            gramx = gramx/gramx.max()*255
            gramy = gramy/gramy.max()*255
            gramx = cv2.cvtColor(gramx.astype(
                np.uint8), cv2.COLOR_GRAY2BGR)
            gramy = cv2.cvtColor(gramy.astype(
                np.uint8), cv2.COLOR_GRAY2BGR)

        gramx = (gramx-gramx.min())/(gramx.max()-gramx.min())*255.0
        gramy = (gramy-gramy.min())/(gramy.max()-gramy.min())*255.0

        if equalize_motiongram:
            gramx = gramx.astype(np.uint8)
            gramx_hsv = cv2.cvtColor(gramx, cv2.COLOR_BGR2HSV)
            gramx_hsv[:, :, 2] = cv2.equalizeHist(gramx_hsv[:, :, 2])
            gramx = cv2.cvtColor(gramx_hsv, cv2.COLOR_HSV2BGR)

            gramy = gramy.astype(np.uint8)
            gramy_hsv = cv2.cvtColor(gramy, cv2.COLOR_BGR2HSV)
            gramy_hsv[:, :, 2] = cv2.equalizeHist(gramy_hsv[:, :, 2])
            gramy = cv2.cvtColor(gramy_hsv, cv2.COLOR_HSV2BGR)

        if target_name_mgx is None:
            target_name_mgx = of+'_mgv.png'
        if target_name_mgy is None:
            target_name_mgy = of+'_mgh.png'
        if not overwrite:
            target_name_mgx = generate_outfilename(target_name_mgx)
            target_name_mgy = generate_outfilename(target_name_mgy)

        if inverted_motiongram:
            cv2.imwrite(target_name_mgx, cv2.bitwise_not(
                gramx.astype(np.uint8)))
            cv2.imwrite(target_name_mgy, cv2.bitwise_not(
                gramy.astype(np.uint8)))
        else:
            cv2.imwrite(target_name_mgx, gramx.astype(np.uint8))
            cv2.imwrite(target_name_mgy, gramy.astype(np.uint8))

        # save rendered motiongrams as MgImages into parent MgVideo
        self.motiongram_x = MgImage(target_name_mgx)
        self.motiongram_y = MgImage(target_name_mgy)

    if save_data:
        # load time
        for idx, item in enumerate(time_files):
            if idx == 0:
                time = np.load(item)
            else:
                last_time = time[-1]
                time = np.append(time, np.load(item)[1:] + last_time)

        # load qom
        for idx, item in enumerate(qom_files):
            if idx == 0:
                qom = np.load(item)
            else:
                qom = np.append(qom, np.load(item)[1:])

        # load com
        for idx, item in enumerate(com_files):
            if idx == 0:
                com = np.load(item)
            else:
                com = np.append(com, np.load(item)[1:], axis=0)

        save_txt(of, time, com, qom, self.width, self.height, data_format,
                 target_name_data=target_name_data, overwrite=overwrite)

    if save_plot:
        if not save_data:
            # load qom
            for idx, item in enumerate(qom_files):
                if idx == 0:
                    qom = np.load(item)
                else:
                    qom = np.append(qom, np.load(item)[1:])

            # load com
            for idx, item in enumerate(com_files):
                if idx == 0:
                    com = np.load(item)
                else:
                    com = np.append(com, np.load(item)[1:], axis=0)

        if title is None:
            title = os.path.basename(of + fex)
        # save plot as an MgImage at motion_plot for parent MgVideo
        self.motion_plot = MgImage(save_analysis(of, self.fps, com, qom, self.width,
                                   self.height, unit, title, target_name_plot=target_name_plot, overwrite=overwrite))

    if save_video:
        if target_name_video is None:
            target_name_video = of + '_motion' + fex
        # enforce avi
        else:
            target_name_video = os.path.splitext(target_name_video)[0] + fex
        if not overwrite:
            target_name_video = generate_outfilename(target_name_video)

        # print("stitching video...")
        concatenated = concat_videos(video_files, pb_prefix='Cleanup')
        # print("moving...")
        os.replace(concatenated, target_name_video)
        # print("checking audio...")
        destination_video = target_name_video
        if self.has_audio:
            source_audio = extract_wav(of + fex)
            embed_audio_in_video(source_audio, destination_video)
            os.remove(source_audio)
        # save rendered motion video as the motion_video of the parent MgVideo
        self.motion_video = musicalgestures.MgVideo(
            destination_video, color=self.color, returned_by_process=True)

    # print("Cleanup...")
    shutil.rmtree(temp_folder)
    # print("Removed temp folder.")

    if save_video:
        return self.motion_video
    else:
        return self

subtract

subtract(color=True, filtertype=None, threshold=0.05, blur=False, curves=0.15, use_median=False, kernel_size=5, bg_img=None, bg_color='#000000', target_name=None, overwrite=True)

Renders background subtraction using ffmpeg.

Parameters:

Name	Type	Description	Default
color	bool	If False the input is converted to grayscale at the start of the process. This can significantly reduce render time. Defaults to True.	True
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	None
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	bool	Whether to apply a smartblur ffmpeg filter or not. Defaults to False.	False
curves	int	Apply curves and equalisation threshold filter to subtract the background. Ranges from 0 to 1. Defaults to 0.15.	0.15
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
kernel_size	int	Size of the median filter (if use_median=True) or the erosion filter (if filtertype='blob'). Defaults to 5.	5
bg_img	str	Path to a background image (.png) that needs to be subtracted from the video. If set to None, it uses an average image of all frames in the video. Defaults to None.	None
bg_color	str	Set the background color in the video file in hex value. Defaults to '#000000' (black).	'#000000'
target_name	str	Target output name for the subtracted video. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	MgVideo	A MgVideo pointing to the subtracted video, also stored as subtract_video on the parent MgVideo

Source code in musicalgestures/_subtract.py

def mg_subtract(
        self,
        color=True,
        filtertype=None,
        threshold=0.05,
        blur=False,
        curves=0.15,
        use_median=False,
        kernel_size=5,
        bg_img=None,
        bg_color='#000000',
        target_name=None,
        overwrite=True) -> "musicalgestures.MgVideo":
    """
    Renders background subtraction using ffmpeg. 

    Args:
        color (bool, optional): If False the input is converted to grayscale at the start of the process. This can significantly reduce render time. Defaults to True.
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (bool, optional): Whether to apply a smartblur ffmpeg filter or not. Defaults to False.
        curves (int, optional): Apply curves and equalisation threshold filter to subtract the background. Ranges from 0 to 1. Defaults to 0.15.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        kernel_size (int, optional): Size of the median filter (if `use_median=True`) or the erosion filter (if `filtertype='blob'`). Defaults to 5.
        bg_img (str, optional): Path to a background image (.png) that needs to be subtracted from the video. If set to None, it uses an average image of all frames in the video. Defaults to None.
        bg_color (str, optional): Set the background color in the video file in hex value. Defaults to '#000000' (black). 
        target_name (str, optional): Target output name for the subtracted video. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A MgVideo pointing to the subtracted video, also stored as `subtract_video` on the parent MgVideo
    """

    of, fex = os.path.splitext(self.filename)

    if target_name is None:
        target_name = of + '_subtracted.avi'

    if not overwrite:
        target_name = generate_outfilename(target_name)

    width, height = self.width, self.height

    if bg_img is None:
        # Render an average image of the video file for background subtraction
        bg_img = musicalgestures.MgVideo(self.filename).blend(component_mode='average').filename
    else:
        # Check if background image extension is .png or not
        pass_if_container_is(".png", bg_img)

    # Set input/output and background color to white
    cmd = ['ffmpeg', '-y', '-i', bg_img, '-i', self.filename]
    cmd_end = ['-shortest', '-pix_fmt', 'yuv420p', target_name]
    cmd_filter = f'color={bg_color}:size={width}x{height} [matte];[1:0]'

    # Set color mode
    if color == True:
        pixformat = 'gbrp'
    else:
        pixformat = 'gray'

    cmd_filter += f'format={pixformat}, split[mask][video];[0:0][mask]'

    # Set frame difference
    if filtertype is not None:
        if filtertype.lower() == 'regular':
            cmd_filter += 'blend=all_mode=difference[diff],'
        else:
            cmd_filter += 'blend=all_mode=difference,'
    else:
        cmd_filter += 'blend=all_mode=difference,'

    thresh_color = matplotlib.colors.to_hex([threshold, threshold, threshold])
    thresh_color = '0x' + thresh_color[1:]

    # Set threshold
    if filtertype is not None:
        if filtertype.lower() == 'regular':
            cmd += ['-f', 'lavfi', '-i', f'color={thresh_color},scale={width}:{height}', 
                    '-f', 'lavfi', '-i', f'color=black,scale={width}:{height}']
            cmd_filter += '[1][diff]threshold,'
        elif filtertype.lower() == 'binary':
            cmd += ['-f', 'lavfi', '-i', f'color={thresh_color},scale={width}:{height}', '-f', 'lavfi', '-i',
                    f'color=black,scale={width}:{height}', '-f', 'lavfi', '-i', f'color=white,scale={width}:{height}']
            cmd_filter += ' threshold,'
        elif filtertype.lower() == 'blob':
            # cmd_filter += 'erosion,' # erosion is always 3x3 so we will hack it with a median filter with percentile=0 which will pick minimum values
            cmd_filter += f'median=radius={kernel_size}:percentile=0,'       

    # Set median
    if use_median and (filtertype is None or filtertype.lower() != 'blob'):  # makes no sense to median-filter the eroded video
        cmd_filter += f'median=radius={kernel_size},'

    # Set curves and equalisation filtering to a range of values between 0.1 and 0.9 
    new_curves = (((curves - 0) * (0.8 - 0.1)) / (1 - 0)) + 0.1
    cmd_filter += f"curves=m='0/0 {str(round(new_curves,2))}/0 {str(round(new_curves+0.1,2))}/1 1/1',"

    # Set blur
    if blur:
        cmd_filter += 'format=gray,smartblur=1,smartblur=3,'

    cmd_filter += f'format=gray [mask];[matte][video][mask] maskedmerge, format={pixformat}' 
    cmd_filter = ['-filter_complex', cmd_filter]  
    cmd = cmd + cmd_filter + cmd_end

    ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Subtracting background:', stream=True)

    # Save subtracted video as subtract_video for parent MgVideo. NB: not
    # `self.subtract`, which would overwrite (shadow) the bound method and
    # break any subsequent subtract() call on the same object.
    self.subtract_video = musicalgestures.MgVideo(target_name, color=color, returned_by_process=True)

    return self.subtract_video

ssm

ssm(features='motiongrams', filtertype='Regular', threshold=0.05, blur='None', norm=np.inf, norm_threshold=0.001, cmap='gray_r', use_median=False, kernel_size=5, invert_yaxis=True, combine=False, title=None, target_name=None, overwrite=True)

Compute Self-Similarity Matrix (SSM) by converting the input signal into a suitable feature sequence and comparing each element of the feature sequence with all other elements of the sequence. SSMs can be computed over different input features such as 'motiongrams', 'spectrogram', 'chromagram' and 'tempogram'.

Parameters:

Name	Type	Description	Default
features	str	Defines the type of features on which to compute SSM. Possible to compute SSM on 'motiongrams', 'videograms', 'spectrogram', 'chromagram' and 'tempogram'. Defaults to 'motiongrams'.	'motiongrams'
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.	'Regular'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
blur	str	'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.	'None'
norm	int	Normalize the columns of the feature sequence. Possible to compute Manhattan norm (1), Euclidean norm (2), Minimum norm (-np.inf), Maximum norm (np.inf), etc. Defaults to np.inf.	inf
norm_threshold	float	Only the columns with norm at least norm_threshold are normalized. Defaults to 0.001.	0.001
combine	bool	For 'motiongrams', compute a single SSM from the concatenated horizontal + vertical motiongram features (both axes of motion in one display) and return a single MgImage instead of an MgList of two. Defaults to False.	False
cmap	str	A Colormap instance or registered colormap name. The colormap maps the C values to colors. Defaults to 'gray_r'.	'gray_r'
use_median	bool	If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.	False
kernel_size	int	Size of the median filter (if use_median=True) or the erosion filter (if filtertype='blob'). Defaults to 5.	5
invert_yaxis	bool	Whether to invert the y axis of the SSM. Defaults to True.	True
title	str	Optionally add title to the figure. Possible to set the filename as the title using the string 'filename'. Defaults to None.	None
target_name	[type]	Target output name for the SSM. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
	'MgList | MgImage'	if features='motiongrams':
MgList	'MgList | MgImage'	An MgList pointing to the output SSM images (as MgImages).
	'MgList | MgImage'	else:
MgImage	'MgList | MgImage'	An MgImage to the output SSM.

Source code in musicalgestures/_ssm.py

def mg_ssm(
        self,
        features: str = 'motiongrams',
        filtertype: str = 'Regular',
        threshold: float = 0.05,
        blur: str = 'None',
        norm: int | float = np.inf,
        norm_threshold: float = 0.001,
        cmap: str = 'gray_r',
        use_median: bool = False,
        kernel_size: int = 5,
        invert_yaxis: bool = True,
        combine: bool = False,
        title: str | None = None,
        target_name: str | None = None,
        overwrite: bool = True) -> "MgList | MgImage":
    """
    Compute Self-Similarity Matrix (SSM) by converting the input signal into a suitable feature sequence and comparing each element of the feature sequence with all other elements of the sequence.
    SSMs can be computed over different input features such as 'motiongrams', 'spectrogram', 'chromagram' and 'tempogram'.

    Args:
        features (str, optional): Defines the type of features on which to compute SSM. Possible to compute SSM on 'motiongrams', 'videograms', 'spectrogram', 'chromagram' and 'tempogram'. Defaults to 'motiongrams'.
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. Defaults to 'Regular'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        blur (str, optional): 'Average' to apply a 10px * 10px blurring filter, 'None' otherwise. Defaults to 'None'.
        norm (int, optional): Normalize the columns of the feature sequence. Possible to compute Manhattan norm (1), Euclidean norm (2), Minimum norm (-np.inf), Maximum norm (np.inf), etc. Defaults to np.inf.
        norm_threshold (float, optional): Only the columns with norm at least `norm_threshold` are normalized. Defaults to 0.001.
        combine (bool, optional): For 'motiongrams', compute a single SSM from the concatenated
            horizontal + vertical motiongram features (both axes of motion in one display) and
            return a single MgImage instead of an MgList of two. Defaults to False.
        cmap (str, optional): A Colormap instance or registered colormap name. The colormap maps the C values to colors. Defaults to 'gray_r'.
        use_median (bool, optional): If True the algorithm applies a median filter on the thresholded frame-difference stream. Defaults to False.
        kernel_size (int, optional):  Size of the median filter (if `use_median=True`) or the erosion filter (if `filtertype='blob'`). Defaults to 5.
        invert_yaxis (bool, optional): Whether to invert the y axis of the SSM. Defaults to True.
        title (str, optional): Optionally add title to the figure. Possible to set the filename as the title using the string 'filename'. Defaults to None.
        target_name ([type], optional): Target output name for the SSM. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        # if features='motiongrams':
        MgList: An MgList pointing to the output SSM images (as MgImages).
        # else:
        MgImage: An MgImage to the output SSM.
    """

    # Save figure to png
    if target_name is None:
        target_name = self.of + '_ssm.png'
    else:
        # enforce png
        target_name = os.path.splitext(target_name)[0] + '.png'
    if not overwrite:
        target_name = generate_outfilename(target_name)

    if features == 'motiongrams':
        # Make sure the file is a video file
        if self.__class__.__name__ == 'MgAudio':
            try:
                width, height = get_widthheight(self.filename)
            except:
                print(f'The "{features}" parameter works only on video files. Try "spectrogram", "chromagram" or "tempogram".')
                return

        out_x, out_y = None, None
        target_name_mgx = os.path.splitext(target_name)[0] + '_mgv.png'
        target_name_mgy = os.path.splitext(target_name)[0] + '_mgh.png'

        if not overwrite:
            out_x = generate_outfilename(target_name_mgx)
            out_y = generate_outfilename(target_name_mgy)
        else:
            out_x, out_y = target_name_mgx, target_name_mgy

        mg_motiongrams(
            self,
            filtertype=filtertype,
            threshold=threshold,
            blur=blur,
            use_median=use_median,
            kernel_size=kernel_size,
            inverted_motiongram=False,
            equalize_motiongram=True,
            target_name_mgx=out_x,
            target_name_mgy=out_y,
            overwrite=True)

        # Normalize feature sequence
        X = librosa.util.normalize(self.ssm_fig.data[0].astype('float32'), norm=norm, threshold=norm_threshold)
        Y = librosa.util.normalize(self.ssm_fig.data[1].astype('float32'), norm=norm, threshold=norm_threshold)

        # Combined SSM: stack horizontal + vertical motiongram features per frame so a
        # single self-similarity matrix reflects both axes of motion at once.
        if combine:
            n = min(X.shape[-1], Y.shape[-1])
            XY = np.concatenate([X[..., :n], Y[..., :n]], axis=0)
            XY_ssm = slow_dot(np.transpose(XY), XY, self.length)

            fig, ax = plt.subplots(figsize=(8, 8))
            if title == 'filename':
                ax.set_title('Combined motion SSM: ' + os.path.basename(self.of + self.fex))
            elif title:
                ax.set_title(title)
            else:
                ax.set_title('Combined (horizontal + vertical) motion SSM')
            img = ax.imshow(XY_ssm, aspect='auto', cmap=cmap)
            if invert_yaxis:
                ax.invert_yaxis()
            ax.set_xlabel('Time [frames]')
            ax.set_ylabel('Time [frames]')
            cb_norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
            fig.colorbar(mpl.cm.ScalarMappable(norm=cb_norm, cmap=cmap), ax=ax, aspect=50)
            fig.tight_layout()
            plt.savefig(target_name, format='png', facecolor='white', transparent=False)
            plt.close()
            self.ssm_combined = MgImage(target_name)
            return MgImage(target_name)

        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)
        Y_ssm = slow_dot(np.transpose(Y), Y, self.length)

       # Plotting Self-Similarity Matrices for motiongrams
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Vertical motiongram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        if invert_yaxis:
            ax0.invert_yaxis()
        img0 = ax0.imshow(X, aspect='auto', cmap=cmap)
        fig.colorbar(img0, ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        img1 = ax1.imshow(X_ssm, aspect='auto', cmap=cmap)
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [frames]')
        ax1.set_ylabel('Time [frames]')
        # Normalize colobar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        plt.savefig(out_x, format='png', facecolor='white', transparent=False)
        plt.close()

        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Horizontal motiongram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        ax0.invert_yaxis()
        img0 = ax0.imshow(Y, aspect='auto', cmap=cmap)
        fig.colorbar(img0, ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        img1 = ax1.imshow(Y_ssm, aspect='auto', cmap=cmap)
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [frames]')
        ax1.set_ylabel('Time [frames]')
        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        plt.savefig(out_y, format='png', facecolor='white', transparent=False)
        plt.close()

        # mg_ssm also saves the motiongrams SSM as MgImages to self.motiongram_x and self.motiongram_y of the parent MgVideo
        return MgList(MgImage(out_x), MgImage(out_y))

    elif features == 'videograms':
        # Make sure the file is a video file
        if self.__class__.__name__ == 'MgAudio':
            try:
                width, height = get_widthheight(self.filename)
            except:
                print(f'The "{features}" parameter works only on video files. Try "spectrogram", "chromagram" or "tempogram".')
                return

        out_x, out_y = None, None
        target_name_vgx = os.path.splitext(target_name)[0] + '_vgv.png'
        target_name_vgy = os.path.splitext(target_name)[0] + '_vgh.png'

        if not overwrite:
            out_x = generate_outfilename(target_name_vgx)
            out_y = generate_outfilename(target_name_vgy)
        else:
            out_x, out_y = target_name_vgx, target_name_vgy

        videograms = videograms_ffmpeg(self,
                                       target_name_x=out_x,
                                       target_name_y=out_y,
                                       overwrite=True)

        pb = MgProgressbar(total=self.length, prefix='Rendering self-similarity matrices:')

        # Load videograms and normalize them
        vgx = cv2.cvtColor(cv2.imread(videograms[0].filename), cv2.COLOR_RGB2GRAY)
        vgy = cv2.cvtColor(cv2.imread(videograms[1].filename), cv2.COLOR_RGB2GRAY)

        X = librosa.util.normalize(vgx.astype('float32'), norm=norm, threshold=norm_threshold)
        Y = librosa.util.normalize(vgy.astype('float32'), norm=norm, threshold=norm_threshold)
        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)
        Y_ssm = slow_dot(np.transpose(Y), Y, self.length)

       # Plotting Self-Similarity Matrices for motiongrams
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Vertical videogram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        ax0.invert_yaxis()
        img0 = ax0.imshow(X, aspect='auto', cmap=cmap)
        fig.colorbar(img0, ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        img1 = ax1.imshow(X_ssm, aspect='auto', cmap=cmap)
        ax1.invert_yaxis()
        ax1.set_xlabel('Time [frames]')
        ax1.set_ylabel('Time [frames]')
        # Normalize colobar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        plt.savefig(out_x, format='png', facecolor='white', transparent=False)
        plt.close()

        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Horizontal videogram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        ax0.invert_yaxis()
        img0 = ax0.imshow(Y, aspect='auto', cmap=cmap)
        fig.colorbar(img0, ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        img1 = ax1.imshow(Y_ssm, aspect='auto', cmap=cmap)
        ax1.invert_yaxis()
        ax1.set_xlabel('Time [frames]')
        ax1.set_ylabel('Time [frames]')
        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        self.ssm_fig = MgFigure(figure=None, figure_type='video.ssm', data=(vgx, vgy), layers=None, image=(target_name_vgx, target_name_vgy))

        plt.savefig(out_y, format='png', facecolor='white', transparent=False)
        plt.close()

        # mg_ssm also saves the motiongrams SSM as MgImages to self.motiongram_x and self.motiongram_y of the parent MgVideo
        return MgList(MgImage(out_x), MgImage(out_y))

    elif features == 'spectrogram':
        if not has_audio(self.filename):
            print('The video has no audio track.')
            return

        # ffprobe for the rate and an extracted track for the samples: librosa 1.0
        # dropped the audioread fallback and can no longer read a video container.
        sr = get_samplerate(self.filename)
        x, sr = librosa.load(audio_source(self.filename), sr=sr)
        frame_length = 512
        hop_length = 128
        spectrogram = np.abs(librosa.stft(x, n_fft=frame_length, hop_length=hop_length))

        X, sr_X, formatter = smooth_downsample_feature_sequence(spectrogram, sr/hop_length)
        # Normalize columns of the feature sequence
        X = librosa.util.normalize(X.astype('float32'), norm=norm, threshold=norm_threshold)
        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)
       # Plotting SSM for spectrogram
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Spectrogram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        img0 = librosa.display.specshow(librosa.amplitude_to_db(X, ref=np.max), y_axis='linear', x_axis='time', cmap=cmap, sr=sr, hop_length=hop_length)
        fig.colorbar(img0, ax=ax0, format="%+2.f dB")
        # Format ticks
        ax0.xaxis.set_major_formatter(formatter)
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        left = - (frame_length / sr) / 2
        right = spectrogram.shape[1] * hop_length / sr + (frame_length / sr) / 2

        img1 = ax1.imshow(librosa.amplitude_to_db(X_ssm, ref=np.max), aspect='auto', cmap=cmap, extent=[left,right,right,left])
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [seconds]')
        ax1.set_ylabel('Time [seconds]')
        fig.colorbar(img1, ax=ax1, aspect=50, format="%+2.f dB")
        fig.tight_layout()

        self.ssm_fig = MgFigure(figure=fig, figure_type='audio.ssm', data=X_ssm, layers=None, image=target_name)

        plt.savefig(target_name, format='png', facecolor='white', transparent=False)
        plt.close()

        return MgImage(target_name)

    elif features == 'chromagram':
        if not has_audio(self.filename):
            print('The video has no audio track.')
            return

        # ffprobe for the rate and an extracted track for the samples: librosa 1.0
        # dropped the audioread fallback and can no longer read a video container.
        sr = get_samplerate(self.filename)
        x, sr = librosa.load(audio_source(self.filename), sr=sr)
        frame_length = 512
        hop_length = 128
        spectrogram = np.abs(librosa.stft(x, n_fft=frame_length, hop_length=hop_length))
        chromagram = librosa.feature.chroma_stft(S=spectrogram, sr=sr, hop_length=hop_length, n_fft=frame_length)

        X, sr_X, formatter = smooth_downsample_feature_sequence(chromagram, sr/hop_length)
        # Normalize feature sequence
        X = librosa.util.normalize(X.astype('float32'), norm=norm, threshold=norm_threshold)
        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)

       # Plotting SSM for chromagram
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Chromagram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        img0 = librosa.display.specshow(X, y_axis='chroma', x_axis='time', cmap=cmap, sr=sr, hop_length=hop_length)

        # Format ticks
        ax0.xaxis.set_major_formatter(formatter)
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax0, aspect=15)
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        left = - (frame_length / sr) / 2
        right = chromagram.shape[1] * hop_length / sr + (frame_length / sr) / 2

        img1 = ax1.imshow(X_ssm, aspect='auto', cmap=cmap, extent=[left,right,right,left])
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [seconds]')
        ax1.set_ylabel('Time [seconds]')
        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        self.ssm_fig = MgFigure(figure=fig, figure_type='audio.ssm', data=X_ssm, layers=None, image=target_name)

        plt.savefig(target_name, format='png', facecolor='white', transparent=False)
        plt.close()

        return MgImage(target_name)

    elif features == 'tempogram':
        if not has_audio(self.filename):
            print('The video has no audio track.')
            return

        # ffprobe for the rate and an extracted track for the samples: librosa 1.0
        # dropped the audioread fallback and can no longer read a video container.
        sr = get_samplerate(self.filename)
        x, sr = librosa.load(audio_source(self.filename), sr=sr)
        frame_length = 1024
        hop_length = 512

        oenv = librosa.onset.onset_strength(y=x, sr=sr, hop_length=hop_length)
        tempogram = librosa.feature.tempogram(onset_envelope=oenv, sr=sr, hop_length=hop_length, win_length=frame_length)
        # Estimate the global tempo for display purposes
        tempo = librosa.feature.tempo(onset_envelope=oenv, sr=sr, hop_length=hop_length)[0]

        X, sr_X, formatter = smooth_downsample_feature_sequence(tempogram, sr/hop_length)
        # Normalize feature sequence
        X = librosa.util.normalize(X.astype('float32'), norm=norm, threshold=norm_threshold)
        # Compute SSM using dot product
        X_ssm = slow_dot(np.transpose(X), X, self.length)

       # Plotting SSM for tempogram
        fig = plt.figure(figsize=(8,8))
        gs = gridspec.GridSpec(4, 1)

        ax0 = fig.add_subplot(gs[0])
        if title is None:
            title = ''
        if title == 'filename':
            title = 'Tempogram: ' + os.path.basename(self.of + self.fex)
        ax0.set_title(title)
        img0 = librosa.display.specshow(X, y_axis='tempo', x_axis='time', cmap=cmap, sr=sr, hop_length=hop_length)

        # Normalize colorbar
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax0, aspect=15)
        ax0.axhline(tempo, color='w', linestyle='--', alpha=1, label='Estimated tempo={:g}'.format(tempo))
        ax0.xaxis.set_major_locator(MaxNLocator(8))
        ax0.xaxis.set_major_formatter(formatter)
        ax0.legend(loc='upper right')
        ax0.set_xlabel('')

        ax1 = fig.add_subplot(gs[1:])
        ax1.xaxis.set_major_locator(MaxNLocator(8))
        ax1.yaxis.set_major_locator(MaxNLocator(8))
        left = - (frame_length / sr) / 2
        right = tempogram.shape[1] * hop_length / sr + (frame_length / sr) / 2

        img1 = ax1.imshow(X_ssm, aspect='auto', cmap=cmap, extent=[left,right,right,left])
        if invert_yaxis:
            ax1.invert_yaxis()
        ax1.set_xlabel('Time [seconds]')
        ax1.set_ylabel('Time [seconds]')
        norm = mpl.colors.Normalize(vmin=0, vmax=1.0)
        fig.colorbar(mpl.cm.ScalarMappable(norm=norm, cmap=cmap), ax=ax1, aspect=50)
        fig.tight_layout()

        self.ssm_fig = MgFigure(figure=fig, figure_type='audio.ssm', data=X_ssm, layers=None, image=target_name)

        plt.savefig(target_name, format='png', facecolor='white', transparent=False)
        plt.close()

        return MgImage(target_name)

    else:
        print(f'Unrecognized feature: "{features}". Try "motiongrams", "videograms, "spectrogram", "chromagram" or "tempogram".')

videograms

videograms(target_name_x=None, target_name_y=None, overwrite=True)

Renders horizontal and vertical videograms of the source video using ffmpeg. Averages videoframes by axes, and creates two images of the horizontal-axis and vertical-axis stacks. In these stacks, a single row or column corresponds to a frame from the source video, and the index of the row or column corresponds to the index of the source frame.

Parameters:

Name	Type	Description	Default
target_name_x	str	Target output name for the vertical videogram (the x-axis collapse). Defaults to None (which assumes that the input filename with the suffix "_vgv" should be used).	None
target_name_y	str	Target output name for the horizontal videogram (the y-axis collapse). Defaults to None (which assumes that the input filename with the suffix "_vgh" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgList	'MgList'	An MgList with the MgImage objects referring to the vertical and horizontal videograms respectively.

Source code in musicalgestures/_videograms.py

def videograms_ffmpeg(self, target_name_x: str | None = None, target_name_y: str | None = None, overwrite: bool = True) -> "MgList":
    """
    Renders horizontal and vertical videograms of the source video using ffmpeg. Averages videoframes by axes, 
    and creates two images of the horizontal-axis and vertical-axis stacks. In these stacks, a single row or 
    column corresponds to a frame from the source video, and the index of the row or column corresponds to 
    the index of the source frame.

    Args:
        target_name_x (str, optional): Target output name for the vertical videogram (the x-axis collapse). Defaults to None (which assumes that the input filename with the suffix "_vgv" should be used).
        target_name_y (str, optional): Target output name for the horizontal videogram (the y-axis collapse). Defaults to None (which assumes that the input filename with the suffix "_vgh" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgList: An MgList with the MgImage objects referring to the vertical and horizontal videograms respectively. 
    """

    width, height = get_widthheight(self.filename)
    framecount = get_framecount(self.filename)

    def calc_skipfactor(width, height, framecount):
        """
        Helper function to calculate the necessary frame-skipping to avoid integer overflow. This makes sure that we can succesfully create videograms even on many-hours-long videos as well.

        Args:
            width (int): The width of the video.
            height (int): The height of the video.
            framecount (int): The number of frames in the video.

        Returns:
            list(int, int): The necessary dilation factors to apply on the video for the horizontal and vertical videograms, respectively.
        """

        intmax = 2147483647
        skipfactor_x = int(
            math.ceil(framecount*8 / (intmax / (height+128) - 1024)))
        skipfactor_y = int(
            math.ceil(framecount / (intmax / ((width*8)+1024) - 128)))
        return skipfactor_x, skipfactor_y

    testx, testy = calc_skipfactor(width, height, framecount)

    if testx > 1 or testy > 1:
        necessary_skipfactor = max([testx, testy])
        print(f'{os.path.basename(self.filename)} is too large to process. Applying minimal skipping necessary...')

        shortened_file = skip_frames_ffmpeg(self.filename, skip=necessary_skipfactor-1)
        skip_of = os.path.splitext(shortened_file)[0]
        framecount = get_framecount(shortened_file)
        length = get_length(shortened_file)

        target_name_x = resolve_filename(skip_of, '_vgv.png', target_name_x, overwrite)
        target_name_y = resolve_filename(skip_of, '_vgh.png', target_name_y, overwrite)

        cmd = ['ffmpeg', '-y', '-i', shortened_file, '-vf',
               f'scale=1:{height}:sws_flags=area,normalize,tile={framecount}x1', '-aspect', f'{framecount}:{height}', '-frames', '1', target_name_y]
        ffmpeg_cmd(cmd, length, stream=False, pb_prefix="Rendering horizontal videogram:")

        cmd = ['ffmpeg', '-y', '-i', shortened_file, '-vf',
               f'scale={width}:1:sws_flags=area,normalize,tile=1x{framecount}', '-aspect', f'{width}:{framecount}', '-frames', '1', target_name_x]
        ffmpeg_cmd(cmd, length, stream=False, pb_prefix="Rendering vertical videogram:")

        # save results as MgImages at self.video_gram_x and self.video_gram_y for parent MgObject
        self.videogram_x = MgImage(target_name_x)
        self.videogram_y = MgImage(target_name_y)

        # return MgList([MgImage(target_name_x), MgImage(target_name_y)])
        return MgList(self.videogram_x, self.videogram_y)


    else:
        length = get_length(self.filename)

        target_name_x = resolve_filename(self.of, '_vgv.png', target_name_x, overwrite)
        target_name_y = resolve_filename(self.of, '_vgh.png', target_name_y, overwrite)

        cmd = ['ffmpeg', '-y', '-i', self.filename, '-frames', '1', '-vf',
               f'scale=1:{height}:sws_flags=area,normalize,tile={framecount}x1', '-aspect', f'{framecount}:{height}', target_name_y]
        ffmpeg_cmd(cmd, length, stream=False, pb_prefix="Rendering horizontal videogram:")

        cmd = ['ffmpeg', '-y', '-i', self.filename, '-frames', '1', '-vf',
               f'scale={width}:1:sws_flags=area,normalize,tile=1x{framecount}', '-aspect', f'{width}:{framecount}', target_name_x]
        ffmpeg_cmd(cmd, length, stream=False, pb_prefix="Rendering vertical videogram:")

        # save results as MgImages at self.videogram_x and self.videogram_y for parent MgObject
        self.videogram_x = MgImage(target_name_x)
        self.videogram_y = MgImage(target_name_y)

        # return MgList([MgImage(target_name_x), MgImage(target_name_y)])
        return MgList(self.videogram_x, self.videogram_y)

directograms

directograms(title=None, filtertype='Adaptative', threshold=0.05, kernel_size=5, convert=True, target_name=None, overwrite=True)

Compute a directogram to factor the magnitude of motion into different angles. Each columun of the directogram is computed as the weighted histogram (HISTOGRAM_BINS) of angles for the optical flow of an input frame.

Source: Abe Davis -- Visual Rhythm and Beat (section 4.1)

Parameters:

Name	Type	Description	Default
title	str	Optionally add title to the figure. Defaults to None, which uses 'Directogram' as a title. Defaults to None.	None
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.	'Adaptative'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
kernel_size	int	Size of structuring element. Defaults to 5.	5
convert	bool	If True (default), non-AVI input is first converted to an all-intra MJPEG .avi (cached as self.as_avi) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.	True
target_name	str	Target output name for the directogram. Defaults to None (which assumes that the input filename with the suffix "_dg" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	A MgFigure object referring to the internal figure and its data.

Source code in musicalgestures/_directograms.py

def mg_directograms(self, title: str | None = None, filtertype: str = 'Adaptative', threshold: float = 0.05, kernel_size: int = 5, convert: bool = True, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Compute a directogram to factor the magnitude of motion into different angles.
    Each columun of the directogram is computed as the weighted histogram (HISTOGRAM_BINS) of angles for the optical flow of an input frame.

    Source: Abe Davis -- [Visual Rhythm and Beat](http://www.abedavis.com/files/papers/VisualRhythm_Davis18.pdf) (section 4.1)

    Args:
        title (str, optional): Optionally add title to the figure. Defaults to None, which uses 'Directogram' as a title. Defaults to None.
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        convert (bool, optional): If True (default), non-AVI input is first converted to an all-intra MJPEG `.avi` (cached as `self.as_avi`) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.
        target_name (str, optional): Target output name for the directogram. Defaults to None (which assumes that the input filename with the suffix "_dg" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgFigure: A MgFigure object referring to the internal figure and its data.
    """

    of, fex = os.path.splitext(self.filename)

    if convert and fex != '.avi':
        # first check if there already is a converted version, if not create one and register it to self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex
        filename = of + fex
    else:
        filename = self.filename

    _ensure_numba()  # JIT-compile the directogram kernels on first use

    vidcap = cv2.VideoCapture(filename)
    fps = int(vidcap.get(cv2.CAP_PROP_FPS))
    length = int(vidcap.get(cv2.CAP_PROP_FRAME_COUNT))

    pb = MgProgressbar(total=length, prefix='Rendering directogram:')

    directograms = []
    directogram_times = np.zeros((length-1,))
    ret, frame = vidcap.read()
    prev_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    i = 0

    while vidcap.isOpened():

        ret, frame = vidcap.read()

        if ret == True:
            next_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            if filtertype == 'Adaptative':
                next_frame = cv2.adaptiveThreshold(next_frame, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
            else:
                # Frame Thresholding: apply threshold filter and median filter (of `kernel_size`x`kernel_size`) to the frame.
                next_frame = filter_frame(next_frame, filtertype, threshold, kernel_size)

            # Renders a dense optical flow video of the input video file using `cv2.calcOpticalFlowFarneback()`.
            # The description of the matching parameters are taken from the cv2 documentation.
            optical_flow = cv2.calcOpticalFlowFarneback(prev_frame, next_frame, None, 0.5, 3, 15, 3, 5, 1.2, 0)
            directograms.append(directogram(optical_flow))
            directogram_times[i] = len(directograms) / fps
            prev_frame = next_frame

        else:
            pb.progress(length)
            break

        pb.progress(i)
        i += 1

    vidcap.release()

    # Create and save the figure
    fig, ax = plt.subplots(figsize=(12, 4), dpi=300)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    # add title
    if title is None:
        title = os.path.basename(f'Directogram (filter type: {filtertype})')

    fig.suptitle(title, fontsize=16)

    ax.imshow(np.array(directograms).T, extent=[directogram_times.min(), directogram_times.max(), 
    HISTOGRAM_BINS.min(), HISTOGRAM_BINS.max()], norm=colors.PowerNorm(gamma=1.0/2.0), aspect='auto')

    ax.set_ylabel('Angle [Radians]')
    ax.set_xlabel('Time [Seconds]')

    target_name = resolve_filename(of, '_dg.png', target_name, overwrite)

    plt.savefig(target_name, format='png', transparent=False)
    plt.close()

    # Create MgFigure
    data = {
        "FPS": fps,
        "path": self.of,
        "directogram times": directogram_times,
        "directogram": np.array(directograms),
    }

    mgf = MgFigure(
        figure=fig,
        figure_type='video.directogram',
        data=data,
        layers=None,
        image=target_name)

    return mgf

warp_audiovisual_beats

warp_audiovisual_beats(audio_file, speed=(0.5, 2), data=None, filtertype='Adaptative', threshold=0.05, kernel_size=5, target_name=None, overwrite=True)

Warp audio beats with visual beats (patterns of motion that can be shifted in time to control visual rhythm). Visual beats are warped after computing a directogram which factors the magnitude of motion in the video into different angles.

Source: Abe Davis -- Visual Rhythm and Beat (section 5)

Parameters:

Name	Type	Description	Default
audio_file	str	Path to the audio file.	required
speed	tuple	Speed's change between the audiovisual beats which can be adjusted to slow down or speed up the visual rhythms. Defaults to (0.5,2).	(0.5, 2)
data	array_like	Computed directogram data can be added separately to avoid the directogram processing time (which can be quite long). Defaults to None.	None
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.	'Adaptative'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
kernel_size	int	Size of structuring element. Defaults to 5.	5
target_name	str	Target output name for the directogram. Defaults to None (which assumes that the input filename with the suffix "_dg" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	'musicalgestures.MgVideo'	A MgVideo as warp_audiovisual_beats for parent MgVideo

Source code in musicalgestures/_warp.py

def mg_warp_audiovisual_beats(self, audio_file: str, speed: tuple = (0.5, 2), data=None, filtertype: str = 'Adaptative', threshold: float = 0.05, kernel_size: int = 5, target_name: str | None = None, overwrite: bool = True) -> "musicalgestures.MgVideo":
    """
    Warp audio beats with visual beats (patterns of motion that can be shifted in time to control visual rhythm).
    Visual beats are warped after computing a directogram which factors the magnitude of motion in the video into different angles.

    Source: Abe Davis -- [Visual Rhythm and Beat](http://www.abedavis.com/files/papers/VisualRhythm_Davis18.pdf) (section 5)

    Args:
        audio_file (str): Path to the audio file.
        speed (tuple, optional): Speed's change between the audiovisual beats which can be adjusted to slow down or speed up the visual rhythms. Defaults to (0.5,2).
        data (array_like, optional): Computed directogram data can be added separately to avoid the directogram processing time (which can be quite long). Defaults to None.
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        target_name (str, optional): Target output name for the directogram. Defaults to None (which assumes that the input filename with the suffix "_dg" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A MgVideo as warp_audiovisual_beats for parent MgVideo
    """

    # COMPUTE DIRECTOGRAMS ------------------------------------------------------------------------------------------------------

    if data is None:
        directogram = mg_directograms(self, title=None, filtertype=filtertype, threshold=threshold, kernel_size=kernel_size, target_name=target_name, overwrite=overwrite)
        directograms = directogram.data['directogram']
        fps = directogram.data['FPS']

    else:
        directograms = data
        vidcap = cv2.VideoCapture(self.filename)
        fps = int(vidcap.get(cv2.CAP_PROP_FPS))

    # COMPUTE AUDIO AND VISUAL BEATS --------------------------------------------------------------------------------------------

    pb = MgProgressbar(total=130, prefix='Warping audiovisual beats:')
    pb.progress(0)
    # audio_source is a no-op for a real audio file and extracts the track if a video
    # is passed instead, which librosa 1.0 can no longer open by itself.
    signal, sr = librosa.load(audio_source(audio_file), mono=True)
    pb.progress(5)

    # Compute onset and impact envelopes
    onset_envelopes = librosa.onset.onset_strength(signal, sr=sr)
    pb.progress(10)
    impact_envelopes = impact_envelope(directograms)
    pb.progress(15)

    # Compute beats with librosa
    pb.progress(20)
    audio_beats = librosa.beat.beat_track(onset_envelope=onset_envelopes,sr=sr, hop_length=512, trim=False, units='samples')
    pb.progress(25)
    visual_beats = librosa.beat.beat_track(onset_envelope=impact_envelopes,sr=fps, hop_length=1.0, trim=False, units='samples')

    # WARP AUDIO AND VISUAL BEATS -----------------------------------------------------------------------------------------------

    _ensure_numba()  # JIT-compile beats_diff on first use
    audio_differences = beats_diff(audio_beats[1], signal)
    pb.progress(30)
    visual_differences = beats_diff(visual_beats[1], np.ndarray.flatten(directograms))
    pb.progress(35)

    # Asserting if the arrays have equal shape and elements
    assert np.array_equal(audio_beats[1], np.cumsum(audio_differences[:-1]))
    assert np.array_equal(visual_beats[1], np.cumsum(visual_differences[:-1]))

    pb.progress(40)
    audio_diff_size = audio_differences.size
    audio_differences_sync = []
    visual_differences_sync = []

    # Loop through each audio and visual beat difference index
    pb.progress(45)
    audio_index, visual_index = 0, 0
    audio_diff = audio_differences[audio_index]
    visual_diff = visual_differences[visual_index]

    pb.progress(50)
    while True:

        # Convert beat differences to time
        audio_time = audio_diff / sr
        visual_time = visual_diff / fps

        speed_change = visual_time / audio_time

        # If the visual beat difference is too short, we check the next index
        if speed_change < speed[0]:
            visual_index += 1
            if visual_index == visual_differences.shape[0]:
                break
            visual_diff = visual_differences[visual_index]

        # If the audio beat difference is too short, we check the next index
        elif speed[1] < speed_change:   
            audio_index += 1
            # Iterate continuously over the audio indexes until visual indexes reach the size of the visual differences array
            audio_diff += audio_differences[audio_index % audio_diff_size]

        else:
            audio_index += 1
            visual_index += 1
            audio_differences_sync.append(audio_diff)
            visual_differences_sync.append(visual_diff)
            if visual_index == visual_differences.shape[0]:
                break
            audio_diff = audio_differences[audio_index % audio_diff_size]
            visual_diff = visual_differences[visual_index]

    pb.progress(55)
    audio_beats_sync = np.cumsum(audio_differences_sync[:-1])
    pb.progress(60)
    visual_beats_sync = np.cumsum(visual_differences_sync[:-1])

    # RENDER AUDIOVISUAL BEATS --------------------------------------------------------------------------------------------------
    pb.progress(65)
    of, fex = os.path.splitext(self.filename)

    if target_name is None:
        target_name = of + '_warped.avi'
    else:
        # enforce avi
        target_name = os.path.splitext(target_name)[0] + '.avi'
    if not overwrite:
        target_name = generate_outfilename(target_name)

    pb.progress(70)
    new_length = audio_beats_sync[-1] + 1
    extended_file_name = f'{audio_file[:-4]}_{new_length}.wav'

    pb.progress(75)
    if not os.path.isfile(extended_file_name):
        data, sample_rate = librosa.load(audio_source(audio_file), mono=True)
        old_length = data.shape[0]
        tail = new_length - old_length * (new_length // old_length)
        extended_data = np.hstack(tuple([data] * (new_length // old_length) + [data[:tail]]))
        sf.write(extended_file_name, extended_data, sample_rate)

    pb.progress(80)
    if os.path.isfile(target_name):
        os.remove(target_name)

    pb.progress(85)        
    temp_file_name = of + '_temp.avi'
    filename = of + '.avi'

    pb.progress(90)
    vidcap = cv2.VideoCapture(filename)
    ret, frame = vidcap.read()
    pb.progress(95)
    output_stream = cv2.VideoWriter(temp_file_name, cv2.VideoWriter_fourcc(*'mp4v'), fps, (frame.shape[1], frame.shape[0]))

    pb.progress(100)
    if ret == True:

        # Iterate through each output frame until the final beat is reached
        last_audio_beat, last_visual_beat = 0, 0
        input_frame_index = 0

        for audio_beat, visual_beat in zip(audio_beats_sync, visual_beats_sync):
            # Output time range
            audio_start_time = last_audio_beat / sr
            audio_end_time = audio_beat / sr
            # Input time range
            visual_start_time = last_visual_beat / fps
            visual_end_time = visual_beat / fps
            # Conversion multiplier
            multiplier = (visual_end_time - visual_start_time) / (audio_end_time - audio_start_time)

            # Iterate through every output frame in the current beat range
            output_start_index = int(np.ceil(audio_start_time * fps))
            output_end_index = int(np.floor(audio_end_time * fps))

            for output_index in range(output_start_index, output_end_index + 1):

                output_time = output_index / fps
                input_time = (output_time - audio_start_time) * multiplier + visual_start_time
                input_index = round(input_time * fps)

                while input_frame_index < input_index:
                    ret, frame = vidcap.read()
                    input_frame_index += 1
                output_stream.write(frame)

            last_audio_beat, last_visual_beat = audio_beat, visual_beat

    # Close visual stream
    pb.progress(105)
    output_stream.release()
    pb.progress(110)
    vidcap.release()

    audio_file = extended_file_name

    pb.progress(115)
    cmd = f'ffmpeg -i {temp_file_name} -i {audio_file} -c:v copy -c:a aac -strict experimental -t {visual_beats_sync[-1] / fps} {wrap_str(target_name)}'

    pb.progress(120)
    subprocess.check_call(cmd, shell=True) 
    pb.progress(125)   
    os.remove(temp_file_name)
    pb.progress(130)

    # Save the warped video on the parent MgVideo. Use a distinct attribute name (not the method
    # name) so it doesn't shadow the warp_audiovisual_beats() method on the instance.
    self.warp_video = musicalgestures.MgVideo(target_name, color=self.color, returned_by_process=True)

    return self.warp_video

blur_faces

blur_faces(mask='blur', mask_image=None, mask_scale=1.0, ellipse=True, draw_heatmap=False, neighbours=32, resolution=250, draw_scores=False, save_data=True, data_format='csv', color=(0, 0, 0), use_gpu=False, target_name=None, overwrite=True)

Automatic anonymization of faces in videos. This function works by first detecting all human faces in each video frame and then applying an anonymization filter (blurring, black rectangles or images) on each detected face region.

Credits: centerface.onnx (original) and centerface.py are based on https://github.com/Star-Clouds/centerface (revision 8c39a49), released under MIT license.

Parameters:

Name	Type	Description	Default
mask	str	Mask filter mode for face regions. 'blur' applies a strong gaussian blurring, 'rectangle' draws a solid black box, 'image' replaces the face with a custom image and 'none' does leaves the input unchanged. Defaults to 'blur'.	'blur'
mask_image	str	Anonymization image path which can be used for masking face regions. This can be activated by specifying 'image' in the mask parameter. Defaults to None.	None
mask_scale	float	Scale factor for face masks, to make sure that the masks cover the complete face. Defaults to 1.0.	1.0
ellipse	bool	Mask faces with blurred ellipses. Defaults to True.	True
draw_heatmap	bool	Draw heatmap of the detected faces using the centroid of the face mask. Defaults to False.	False
neighbours	int	Number of neighbours for smoothing the heatmap image. Defaults to 32.	32
resolution	int	Number of pixel resolution for the heatmap visualization. Defaults to 250.	250
draw_scores	bool	Draw detection faceness scores onto outputs (a score between 0 and 1 that roughly corresponds to the detector's confidence that something is a face). Defaults to False.	False
save_data	bool	Whether to save the scaled coordinates of the face mask (time (ms), x1, y1, x2, y2) for each frame to a file. Defaults to True.	True
data_format	str	Specifies format of blur_faces-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, e.g. ['csv', 'txt']. Defaults to 'csv'.	'csv'
color	tuple	Customized color of the rectangle boxes. Defaults to black (0, 0, 0).	(0, 0, 0)
use_gpu	bool	Whether to attempt GPU (CUDA) acceleration for face detection. Falls back to CPU automatically if CUDA is unavailable. Defaults to False.	False
target_name	str	Target output name. Defaults to None (which assumes that the input filename with the suffix "_blurred" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo		A MgVideo as blur_faces for parent MgVideo

Source code in musicalgestures/_blurfaces.py

def mg_blurfaces(self, 
                 mask='blur', 
                 mask_image=None, 
                 mask_scale=1.0, 
                 ellipse=True, 
                 draw_heatmap=False, 
                 neighbours=32, 
                 resolution=250, 
                 draw_scores=False, 
                 save_data=True, 
                 data_format='csv', 
                 color=(0, 0, 0), 
                 use_gpu=False,
                 target_name=None, 
                 overwrite=True):
    """
    Automatic anonymization of faces in videos. 
    This function works by first detecting all human faces in each video frame and then applying an anonymization filter 
    (blurring, black rectangles or images) on each detected face region.

    Credits: `centerface.onnx` (original) and `centerface.py` are based on https://github.com/Star-Clouds/centerface (revision 8c39a49), released under [MIT license](https://github.com/Star-Clouds/CenterFace/blob/36afed/LICENSE).

    Args:
        mask (str, optional): Mask filter mode for face regions. 'blur' applies a strong gaussian blurring, 'rectangle' draws a solid black box, 'image' replaces the face with a custom image and 'none' does leaves the input unchanged. Defaults to 'blur'.
        mask_image (str, optional): Anonymization image path which can be used for masking face regions. This can be activated by specifying 'image' in the mask parameter. Defaults to None.
        mask_scale (float, optional): Scale factor for face masks, to make sure that the masks cover the complete face. Defaults to 1.0.
        ellipse (bool, optional): Mask faces with blurred ellipses. Defaults to True.
        draw_heatmap (bool, optional): Draw heatmap of the detected faces using the centroid of the face mask. Defaults to False.
        neighbours (int, optional): Number of neighbours for smoothing the heatmap image. Defaults to 32.
        resolution (int, optional): Number of pixel resolution for the heatmap visualization. Defaults to 250.
        draw_scores (bool, optional): Draw detection faceness scores onto outputs (a score between 0 and 1 that roughly corresponds to the detector's confidence that something is a face). Defaults to False.
        save_data (bool, optional): Whether to save the scaled coordinates of the face mask (time (ms), x1, y1, x2, y2) for each frame to a file. Defaults to True.
        data_format (str, optional): Specifies format of blur_faces-data. Accepted values are 'csv', 'tsv' and 'txt'. For multiple output formats, use list, e.g. ['csv', 'txt']. Defaults to 'csv'.
        color (tuple, optional): Customized color of the rectangle boxes. Defaults to black (0, 0, 0).
        use_gpu (bool, optional): Whether to attempt GPU (CUDA) acceleration for face detection. Falls back to CPU automatically if CUDA is unavailable. Defaults to False.
        target_name (str, optional): Target output name. Defaults to None (which assumes that the input filename with the suffix "_blurred" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A MgVideo as blur_faces for parent MgVideo
    """

    of, fex = os.path.splitext(self.filename)

    if target_name is None:
        # keep the source container (e.g. .mp4) by default
        target_name = of + '_blurred' + fex
    if not overwrite:
        target_name = generate_outfilename(target_name)
    if os.path.isfile(target_name):
        os.remove(target_name)

    pb = MgProgressbar(total=self.length, prefix='Blurring faces:')

    # Create an instance of the CenterFace class
    centerface = CenterFace(use_gpu=use_gpu)
    # Output is written through an FFmpeg pipe (libx264) for a small, portable file
    video_out = None
    # Create an empty list to append the mask coordinates
    data = []

    # Define ffmpeg command start and end
    cmd = ['ffmpeg', '-y', '-i', self.filename]
    process = ffmpeg_cmd(cmd, total_time=self.length, pipe='read')

    i = 0

    while True:
        # Read frame-by-frame
        if self.color:
            out = process.stdout.read(self.width*self.height*3)
        else:
            out = process.stdout.read(self.width*self.height)
        if out == b'':
            pb.progress(self.length)
            break

        # Transform the bytes read into a numpy array
        frame = np.frombuffer(out, dtype=np.uint8).reshape([self.height, self.width, 3]) # height, width, channels
        frame = frame.copy() # copy frame for writing it

        h, w = frame.shape[:2]
        dets, lms = centerface(frame, h, w, threshold=0.2)

        for x, det in enumerate(dets):
            boxes, score = det[:4], det[4]
            x1, y1, x2, y2 = boxes.astype(int)
            x1, y1, x2, y2 = scaling_mask(x1, y1, x2, y2, mask_scale)
            # Clip bounding boxes coordinates to valid frame region
            y1, y2 = max(0, y1), min(frame.shape[0] - 1, y2)
            x1, x2 = max(0, x1), min(frame.shape[1] - 1, x2)

            # Mask faces with rectangles
            if mask == 'rectangle':
                # Color is set to black by default but can be changed using the color parameter.
                cv2.rectangle(frame, (x1, y1), (x2, y2), color, -1)

            # Mask faces with blurred rectangles
            elif mask == 'blur':
                bf = 2  # blur factor (number of pixels in each dimension that the face will be reduced to)
                blurred_box = cv2.blur(frame[y1:y2, x1:x2], (abs(x2 - x1) // bf, abs(y2 - y1) // bf))
                # Mask faces with blurred ellipses
                if ellipse:
                    roibox = frame[y1:y2, x1:x2]
                    # Get y and x coordinate lists of the bounding ellipse
                    ey, ex = skimage.draw.ellipse((y2 - y1) // 2, (x2 - x1) // 2, (y2 - y1) // 2, (x2 - x1) // 2)
                    roibox[ey, ex] = blurred_box[ey, ex]
                    frame[y1:y2, x1:x2] = roibox
                else:
                    frame[y1:y2, x1:x2] = blurred_box

            # Mask faces with an image
            elif mask == 'image':
                target_size = (x2 - x1, y2 - y1)
                # Reading image with opencv
                mask_image = cv2.imread(mask_image, cv2.IMREAD_UNCHANGED)
                # Resizing with the target size
                resized_mask_image = cv2.resize(mask_image, target_size)
                if mask_image.shape[2] == 3:  # RGB
                    frame[y1:y2, x1:x2] = resized_mask_image
                elif mask_image.shape[2] == 4:  # RGBA
                    frame[y1:y2, x1:x2] = frame[y1:y2, x1:x2] * (1 - resized_mask_image[:, :, 3:] / 255) + resized_mask_image[:, :, :3] * (resized_mask_image[:, :, 3:] / 255)  

            # Mask nothing
            elif mask == 'none':
                pass

            # Draw heatmap of the detected faces using the centroid of the face mask.
            if draw_heatmap or save_data:
                time = frame2ms(i, self.fps)
                data.append([time, x1, y1, x2, y2])

            # Draw the faceness score (between 0 and 1) that roughly corresponds to the detector's confidence that something is a face.
            if draw_scores:
                cv2.putText(frame, f'{score:.2f}', (x1 + 0, y1 - 20), cv2.FONT_HERSHEY_DUPLEX, 0.5, (0, 255, 0))

        if video_out is None:
            cmd = ['ffmpeg', '-y', '-s', '{}x{}'.format(frame.shape[1], frame.shape[0]),
                   '-r', str(self.fps), '-f', 'rawvideo', '-pix_fmt', 'bgr24',
                   '-vcodec', 'rawvideo', '-i', '-', '-vcodec', 'libx264',
                   '-pix_fmt', 'yuv420p', target_name]
            video_out = ffmpeg_cmd(cmd, total_time=self.length, pipe='write')
        video_out.stdin.write(frame.astype(np.uint8).tobytes())

        # Flush the buffer
        process.stdout.flush()
        pb.progress(i)
        i += 1

    # Terminate the process
    process.terminate()
    if video_out is not None:
        video_out.stdin.close()
        video_out.wait()

    if self.has_audio:
        # Embed audio in the video file
        source_audio = extract_wav(of + fex)
        embed_audio_in_video(source_audio, target_name)
        os.remove(source_audio)

    # Save warped video as blur_faces for parent MgVideo
    # we have to do this here since we are not using mg_blurfaces (that would normally save the result itself)
    self.blur_faces_video = musicalgestures.MgVideo(target_name, color=self.color, returned_by_process=True)

    def save_txt(of, data, data_format, target_name=target_name, overwrite=overwrite):
        """
        Helper function to export pose estimation data as textfile(s).
        """
        def save_single_file(of, data, data_format, target_name=target_name, overwrite=overwrite):
            """
            Helper function to export pose estimation data as a textfile using pandas.
            """

            headers = ['time (ms)', 'x1', 'y1', 'x2', 'y2']
            data_format = data_format.lower()

            df = pd.DataFrame(data=data, columns=headers)

            if data_format == "tsv":
                if target_name is None:
                    target_name = of + '.tsv'
                else:
                    # take name, but enforce tsv
                    target_name = os.path.splitext(target_name)[0] + '.tsv'
                if not overwrite:
                    target_name = generate_outfilename(target_name)

                with open(target_name, 'wb') as f:
                    head_str = ''
                    for head in headers:
                        head_str += head + '\t'
                    head_str += '\n'
                    f.write(head_str.encode())
                    fmt_list = ['%.0f' for item in range(len(headers))]
                    np.savetxt(f, df.values, delimiter='\t', fmt=fmt_list)

            elif data_format == "csv":
                if target_name is None:
                    target_name = of + '.csv'
                else:
                    # take name, but enforce csv
                    target_name = os.path.splitext(target_name)[0] + '.csv'
                if not overwrite:
                    target_name = generate_outfilename(target_name)
                df.to_csv(target_name, index=None)

            elif data_format == "txt":
                if target_name is None:
                    target_name = of + '.txt'
                else:
                    # take name, but enforce txt
                    target_name = os.path.splitext(target_name)[0] + '.txt'
                if not overwrite:
                    target_name = generate_outfilename(target_name)

                with open(target_name, 'wb') as f:
                    head_str = ''
                    for head in headers:
                        head_str += head + ' '
                    head_str += '\n'
                    f.write(head_str.encode())
                    fmt_list = ['%.0f' for item in range(len(headers))]
                    np.savetxt(f, df.values, delimiter=' ', fmt=fmt_list)

            elif data_format not in ["tsv", "csv", "txt"]:
                print(f"Invalid data format: '{data_format}'.\nFalling back to '.csv'.")
                save_single_file(of, data, "csv", target_name=target_name, overwrite=overwrite)

        if type(data_format) == str:
            save_single_file(of, data, data_format, target_name=target_name, overwrite=overwrite)

        elif type(data_format) == list:
            if all([item.lower() in ["csv", "tsv", "txt"] for item in data_format]):
                data_format = list(set(data_format))
                [save_single_file(of, data, item, target_name=target_name, overwrite=overwrite)
                 for item in data_format]
            else:
                print(f"Unsupported formats in {data_format}.\nFalling back to '.csv'.")
                save_single_file(of, data, "csv", target_name=target_name, overwrite=overwrite)

    if save_data:  
        save_txt(of, data, data_format, target_name=target_name, overwrite=overwrite)  
        return self.blur_faces_video

    if draw_heatmap:
        target_name = os.path.splitext(target_name)[0] + '.png'
        if not overwrite:
            target_name = generate_outfilename(target_name)

        # Approximately update font and plot size with frame size
        plt.rcParams.update({'font.size': int((self.height/self.fps))})  
        fig, ax = plt.subplots(figsize=(int(self.width/self.fps), int(self.height/self.fps))) 
        # make sure background is white
        fig.patch.set_facecolor('white')
        fig.patch.set_alpha(1)  

        center_x, center_y = centroid_mask(np.asarray(data))
        im, extent = nearest_neighbours(center_x, center_y, self.width, self.height, resolution, neighbours)

        ax.imshow(im, extent=extent, cmap=cm.jet)
        ax.set_title(f"Heatmap of face detection (neighbours={neighbours})")
        ax.set_xlabel("Video width (pixels)")
        ax.set_ylabel("Video height (pixels)")
        ax.set_xlim(extent[0], extent[1])
        ax.set_ylim(extent[2], extent[3])

        divider = make_axes_locatable(ax)
        cax = divider.append_axes("right", size="5%", pad="3%")
        normalizer = mpl.colors.Normalize(vmin=0, vmax=1)
        fig.colorbar(mpl.cm.ScalarMappable(norm=normalizer, cmap=cm.jet), cax=cax)

        plt.tight_layout()
        plt.savefig(target_name, format='png', transparent=False)
        plt.close()

        return MgImage(target_name)

    else:
        return self.blur_faces_video

impacts

impacts(title=None, detection=True, local_mean=0.1, local_maxima=0.15, filtertype='Adaptative', threshold=0.05, kernel_size=5, convert=True, target_name=None, overwrite=True)

Compute a visual analogue of an onset envelope, aslo known as an impact envelope (Abe Davis). This is computed by summing over positive entries in the columns of the directogram. This gives an impact envelope with precisely the same form as an onset envelope. To account for large outlying spikes that sometimes happen at shot boundaries (i.e., cuts), the 99th percentile of the impact envelope values are clipped to the 98th percentile. Then, the impact envelopes are normalized by their maximum to make calculations more consistent across video resolutions. Fianlly, the local mean of the impact envelopes are calculated using a 0.1-second window, and local maxima using a 0.15-second window. Impacts are defined as local maxima that are above their local mean by at least 10% of the envelope’s global maximum.

Source: Abe Davis -- Visual Rhythm and Beat (section 4.2 and 4.3)

Parameters:

Name	Type	Description	Default
title	str	Optionally add title to the figure. Defaults to None, which uses 'Directogram' as a title. Defaults to None.	None
detection	bool	Whether to allow the detection of impacts based on local mean and local maxima or not.	True
local_mean	float	Size of the local mean window in seconds which reduces the amount of intensity variation between one impact and the next.	0.1
local_maxima	float	Size of the local maxima window in seconds for the impact envelopes	0.15
filtertype	str	'Regular' turns all values below threshold to 0. 'Binary' turns all values below threshold to 0, above threshold to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.	'Adaptative'
threshold	float	Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.	0.05
kernel_size	int	Size of structuring element. Defaults to 5.	5
convert	bool	If True (default), non-AVI input is first converted to an all-intra MJPEG .avi (cached as self.as_avi) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.	True
target_name	str	Target output name for the impacts figure. Defaults to None (which assumes that the input filename with the suffix "_impacts" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	An MgFigure object referring to the internal figure and its data.

Source code in musicalgestures/_impacts.py

def mg_impacts(self, title: str | None = None, detection: bool = True, local_mean: float = 0.1, local_maxima: float = 0.15, filtertype: str = 'Adaptative', threshold: float = 0.05, kernel_size: int = 5, convert: bool = True, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Compute a visual analogue of an onset envelope, aslo known as an impact envelope (Abe Davis).
    This is computed by summing over positive entries in the columns of the directogram. This gives an impact envelope with precisely the same
    form as an onset envelope. To account for large outlying spikes that sometimes happen at shot boundaries (i.e., cuts), the 99th percentile
    of the impact envelope values are clipped to the 98th percentile. Then, the impact envelopes are normalized by their maximum to make calculations
    more consistent across video resolutions. Fianlly, the local mean of the impact envelopes are calculated using a 0.1-second window, and local maxima
    using a 0.15-second window. Impacts are defined as local maxima that are above their local mean by at least 10% of the envelope’s global maximum.

    Source: Abe Davis -- [Visual Rhythm and Beat](http://www.abedavis.com/files/papers/VisualRhythm_Davis18.pdf) (section 4.2 and 4.3)

    Args:
        title (str, optional): Optionally add title to the figure. Defaults to None, which uses 'Directogram' as a title. Defaults to None.
        detection (bool, optional): Whether to allow the detection of impacts based on local mean and local maxima or not.
        local_mean (float, optional): Size of the local mean window in seconds which reduces the amount of intensity variation between one impact and the next.
        local_maxima (float, optional): Size of the local maxima window in seconds for the impact envelopes
        filtertype (str, optional): 'Regular' turns all values below `threshold` to 0. 'Binary' turns all values below `threshold` to 0, above `threshold` to 1. 'Blob' removes individual pixels with erosion method. 'Adaptative' perform adaptative threshold as the weighted sum of 11 neighborhood pixels where weights are a Gaussian window. Defaults to 'Adaptative'.
        threshold (float, optional): Eliminates pixel values less than given threshold. Ranges from 0 to 1. Defaults to 0.05.
        kernel_size (int, optional): Size of structuring element. Defaults to 5.
        convert (bool, optional): If True (default), non-AVI input is first converted to an all-intra MJPEG `.avi` (cached as `self.as_avi`) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.
        target_name (str, optional): Target output name for the impacts figure. Defaults to None (which assumes that the input filename with the suffix "_impacts" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgFigure: An MgFigure object referring to the internal figure and its data.
    """

    of, fex = os.path.splitext(self.filename)

    if convert and fex != '.avi':
        # first check if there already is a converted version, if not create one and register it to self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex
        filename = of + fex
    else:
        filename = self.filename

    _directograms._ensure_numba()  # JIT-compile the directogram kernels on first use
    _ensure_numba()                # JIT-compile impact_detection on first use

    vidcap = cv2.VideoCapture(filename)
    fps = int(vidcap.get(cv2.CAP_PROP_FPS))
    width = int(vidcap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(vidcap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    length = int(vidcap.get(cv2.CAP_PROP_FRAME_COUNT))

    pb = MgProgressbar(total=length, prefix='Rendering impact envelopes:')

    directograms = []
    directogram_times = []
    ret, frame = vidcap.read()
    prev_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    i = 0

    while vidcap.isOpened():

        ret, frame = vidcap.read()

        if ret == True:
            next_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            if filtertype == 'Adaptative':
                next_frame = cv2.adaptiveThreshold(
                    next_frame, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
            else:
                # Frame Thresholding: apply threshold filter and median filter (of `kernel_size`x`kernel_size`) to the frame.
                next_frame = filter_frame(next_frame, filtertype, threshold, kernel_size)

            # Renders a dense optical flow video of the input video file using `cv2.calcOpticalFlowFarneback()`.
            # The description of the matching parameters are taken from the cv2 documentation.
            optical_flow = cv2.calcOpticalFlowFarneback(
                prev_frame, next_frame, None, 0.5, 3, 15, 3, 5, 1.2, 0)
            directograms.append(_directograms.directogram(optical_flow))
            directogram_times.append(len(directograms) / fps) 
            prev_frame = next_frame

        else:
            pb.progress(length)
            break

        pb.progress(i)
        i += 1

    vidcap.release()

    # Compute impact envelopes and impact detection
    impact_envelopes = impact_envelope(np.array(directograms))
    impacts = np.array(impact_detection(impact_envelopes, np.array(directogram_times), fps, local_mean=local_mean, local_maxima=local_maxima)) / fps # convert to seconds

    fig, ax = plt.subplots(figsize=(12, 4), dpi=300)

    # make sure background is white
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    # add title
    if title is None:
        title = os.path.basename(
            f'Impact Envelopes (filter type: {filtertype})')

    fig.suptitle(title, fontsize=16)

    ax.plot(directogram_times, impact_envelopes)
    ax.set_xlabel('Time [Seconds]')
    ax.set_yticks([])
    ax.margins(x=0)

    if detection:
        ax.vlines(impacts, 0, max(impact_envelopes), colors='red', linestyles='dashed',
                  label=f'Impact Detection\nLocal mean: {local_mean}\nLocal maxima: {local_maxima}')
        ax.legend(loc='upper right')

    fig.tight_layout()

    target_name = resolve_filename(of, '_impact.png', target_name, overwrite)

    plt.savefig(target_name, format='png', transparent=False)
    plt.close()

    # create MgFigure
    data = {
        "FPS": fps,
        "path": self.of,
        "impact times": directogram_times,
        "impact envelopes": impact_envelopes,
    }

    mgf = MgFigure(
        figure=fig,
        figure_type='video.impacts',
        data=data,
        layers=None,
        image=target_name)

    return mgf

grid

grid(height=300, rows=3, columns=3, padding=0, margin=0, target_name=None, overwrite=True, return_array=False)

Generates frame strip video preview using ffmpeg.

Parameters:

Name	Type	Description	Default
height	int	Frame height, width is adjusted automatically to keep the correct aspect ratio. Defaults to 300.	300
rows	int	Number of rows of the grid. Defaults to 3.	3
columns	int	Number of columns of the grid. Defaults to 3.	3
padding	int	Padding size between the frames. Defaults to 0.	0
margin	int	Margin size for the grid. Defaults to 0.	0
target_name	[type]	Target output name for the grid image. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True
return_array	bool	Whether to return an array of not. If set to False the function writes the grid image to disk. Defaults to False.	False

Returns:

Name	Type	Description
MgImage		An MgImage object referring to the internal grid image.

Source code in musicalgestures/_grid.py

def mg_grid(self, height=300, rows=3, columns=3, padding=0, margin=0, target_name=None, overwrite=True, return_array=False):
    """
    Generates frame strip video preview using ffmpeg.

    Args:
        height (int, optional): Frame height, width is adjusted automatically to keep the correct aspect ratio. Defaults to 300.
        rows (int, optional): Number of rows of the grid. Defaults to 3.
        columns (int, optional): Number of columns of the grid. Defaults to 3.
        padding (int, optional): Padding size between the frames. Defaults to 0.
        margin (int, optional): Margin size for the grid. Defaults to 0.
        target_name ([type], optional): Target output name for the grid image. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.
        return_array (bool, optional): Whether to return an array of not. If set to False the function writes the grid image to disk. Defaults to False.

    Returns:
        MgImage: An MgImage object referring to the internal grid image.
    """

    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_grid.png', target_name, overwrite)

    # Get the number of frames
    cap = cv2.VideoCapture(self.filename)
    nb_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    nth_frame = int(nb_frames / (rows*columns))

    # Define the grid specifications
    width = int((float(self.width) / self.height) * height)
    grid = rf"select=not(mod(n\,{nth_frame})),scale={width}:{height},tile={columns}x{rows}:padding={padding}:margin={margin}"

    # Declare the ffmpeg commands
    if return_array:
        cmd = ['ffmpeg', '-y', '-i', self.filename, '-frames', '1', '-q:v', '0', '-vf', grid]
        process = ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering video frame grid:', pipe='load')

        # Convert bytes to array and convert from BGR to RGB
        array = np.frombuffer(process.stdout, dtype=np.uint8).reshape([height*rows, int(width*columns), 3])[...,::-1] 

        return array
    else:
        cmd = ['ffmpeg', '-i', self.filename, '-y', '-frames', '1', '-q:v', '0', '-vf', grid, target_name]
        ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering video frame grid:')
        # Initialize the MgImage object
        img = MgImage(target_name)

        return img

resample

resample(fps=None, speed=None, skip=None, target_name=None, overwrite=True)

Resample the (already loaded) video and return a new MgVideo, leaving the original object untouched.

Three independent, combinable operations:

• fps: retime to a target frame rate using FFmpeg's fps filter — duration-preserving (frames are dropped/duplicated to hit the rate), e.g. 30 → 25 fps.
• speed: change playback speed by a factor (>1 faster/shorter, <1 slower/longer); the video is retimed with setpts and the audio with atempo so they stay in sync.
• skip: integer frame decimation — discard skip frames for every one kept (this also shortens/speeds up the clip), matching the loader's skip parameter.

When more than one is given they are applied in order: skip → speed/fps.

Parameters:

Name	Type	Description	Default
fps	float	Target frame rate (duration-preserving). Defaults to None.	None
speed	float	Playback-speed factor. Defaults to None.	None
skip	int	Discard skip frames for every one kept. Defaults to None.	None
target_name	str	Output name. Defaults to None (input filename + "_resampled").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	MgVideo	a new MgVideo pointing to the resampled file.

Source code in musicalgestures/_videoadjust.py

def mg_resample(self, fps=None, speed=None, skip=None, target_name=None, overwrite=True) -> "musicalgestures.MgVideo":
    """
    Resample the (already loaded) video and return a **new** MgVideo, leaving the original
    object untouched.

    Three independent, combinable operations:

    * ``fps``: retime to a target frame rate using FFmpeg's ``fps`` filter — **duration-preserving**
      (frames are dropped/duplicated to hit the rate), e.g. 30 → 25 fps.
    * ``speed``: change playback speed by a factor (>1 faster/shorter, <1 slower/longer); the video
      is retimed with ``setpts`` and the audio with ``atempo`` so they stay in sync.
    * ``skip``: integer frame decimation — discard ``skip`` frames for every one kept (this also
      shortens/speeds up the clip), matching the loader's ``skip`` parameter.

    When more than one is given they are applied in order: ``skip`` → ``speed``/``fps``.

    Args:
        fps (float, optional): Target frame rate (duration-preserving). Defaults to None.
        speed (float, optional): Playback-speed factor. Defaults to None.
        skip (int, optional): Discard ``skip`` frames for every one kept. Defaults to None.
        target_name (str, optional): Output name. Defaults to None (input filename + "_resampled").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgVideo: a new MgVideo pointing to the resampled file.
    """
    import musicalgestures

    if fps is None and speed is None and not skip:
        raise ValueError("Provide at least one of fps, speed, or skip.")
    if speed is not None and speed <= 0:
        raise ValueError("speed must be a positive factor (e.g. 2.0 = twice as fast).")
    if fps is not None and fps <= 0:
        raise ValueError("fps must be a positive number.")

    source = self.filename

    # 1) Integer frame decimation (also speeds up) — reuse the tested helper.
    if skip:
        source = skip_frames_ffmpeg(source, int(skip), overwrite=overwrite)

    # 2) Speed and/or frame-rate retime in a single FFmpeg pass.
    final = source
    if speed is not None or fps is not None:
        of, fex = os.path.splitext(source)
        if target_name is None:
            out = of + '_resampled' + fex
        else:
            out = os.path.splitext(target_name)[0] + fex
        if not overwrite:
            out = generate_outfilename(out)

        vfilters = []
        if speed is not None and speed != 1:
            vfilters.append(f'setpts=PTS/{speed:.6g}')
        if fps is not None:
            vfilters.append(f'fps={fps:.6g}')
        vf = ','.join(vfilters)

        if speed is not None and speed != 1 and has_audio(source):
            # Retime audio too so it stays in sync with the sped-up/slowed-down video.
            atempo_chain = _build_atempo_chain(speed)
            cmd = ['ffmpeg', '-y', '-i', source, '-filter_complex',
                   f'[0:v]{vf}[v];[0:a]{atempo_chain}[a]', '-map', '[v]', '-map', '[a]',
                   '-q:v', '3', '-shortest', out]
        else:
            # fps-only (or no audio): -vf retimes the video and passes audio through unchanged.
            cmd = ['ffmpeg', '-y', '-i', source, '-vf', vf, '-q:v', '3', out]

        ffmpeg_cmd(cmd, get_length(source), pb_prefix='Resampling:')
        final = out

    return musicalgestures.MgVideo(final, color=self.color, returned_by_process=True)

save_analysis

save_analysis(of, fps, aom, com, qom, motion_analysis, audio_descriptors, width, height, unit, title, target_name_plot, overwrite)

Helper function to plot the motion data using matplotlib.

Source code in musicalgestures/_motionvideo.py

def save_analysis(of, fps, aom, com, qom, motion_analysis, audio_descriptors, width, height, unit, title, target_name_plot, overwrite):
    """
    Helper function to plot the motion data using matplotlib.
    """
    plt.rc('text', usetex=False)
    fig = plt.figure(figsize=(12, 10), dpi=300)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)
    # add title
    fig.suptitle(title, fontsize=16)

    if motion_analysis.lower() == 'all':
        gs = gridspec.GridSpec(3, 2)
    elif motion_analysis.lower() == 'qom':
        gs = gridspec.GridSpec(2, 2)
    else:
        gs = gridspec.GridSpec(1, 2)

    # Audio descriptors
    if audio_descriptors:
        plt.close(fig)  # discard the figure created above; build a taller one instead
        fig = plt.figure(figsize=(12, 16), dpi=300)
        fig.patch.set_facecolor('white')
        fig.patch.set_alpha(1)
        # add title
        fig.suptitle(title, fontsize=16)
        descriptors = audio_descriptors.audio.descriptors(autoshow=False).data

        if motion_analysis.lower() == 'all':
            gs = gridspec.GridSpec(6, 2)
        elif motion_analysis.lower() == 'qom':
            gs = gridspec.GridSpec(5, 2)
        else:
            gs = gridspec.GridSpec(4, 2)

    # Centroid of motion (CoM)
    if motion_analysis.lower() == 'com':
        ax0 = fig.add_subplot(gs[0, :])
        ax0.scatter(com[:, 0]/width, com[:, 1]/height, s=2)
        ax0.set_xlim((0, 1))
        ax0.set_ylim((0, 1))
        ax0.set_xlabel('Pixels normalized')
        ax0.set_ylabel('Pixels normalized')
        ax0.set_title('Centroid of motion (CoM)')

    # Area of motion (AoM)
    if motion_analysis.lower() == 'aom':
        ax0 = fig.add_subplot(gs[0, :])
        ax0.scatter(aom[:, 0], aom[:, 1], c='C0', s=2)
        ax0.scatter(aom[:, 2], aom[:, 3], c='C0', s=2)
        ax0.set_xlim((0, 1))
        ax0.set_ylim((0, 1))
        ax0.set_xlabel('Pixels normalized')
        ax0.set_ylabel('Pixels normalized')
        ax0.set_title('Area of motion (AoM)')

    # Quantity of motion (QoM)
    def adjacent_values(vals, q1, q3):
        upper_adjacent_value = q3 + (q3 - q1) * 1.5
        upper_adjacent_value = np.clip(upper_adjacent_value, q3, vals[-1])

        lower_adjacent_value = q1 - (q3 - q1) * 1.5
        lower_adjacent_value = np.clip(lower_adjacent_value, vals[0], q1)
        return lower_adjacent_value, upper_adjacent_value

    if motion_analysis.lower() == 'qom':
        ax0 = fig.add_subplot(gs[0, :])
        ax0.set_title('Quantity of motion (QoM)')
        ax0.violinplot([qom[1:]/(max(qom[1:]))], showmeans=False, showmedians=True, showextrema=True, vert=False)

        quartile1, medians, quartile3 = np.percentile([qom[1:]/(max(qom[1:]))], [25, 50, 75], axis=1)
        whiskers = np.array([adjacent_values(sorted_array, q1, q3) for sorted_array, q1, q3 in zip([qom[1:]/(max(qom[1:]))], quartile1, quartile3)])
        whiskers_min, whiskers_max = whiskers[:, 0], whiskers[:, 1]

        inds = np.arange(1, len(medians) + 1)
        ax0.scatter(medians, inds, marker='o', color='white', s=60, zorder=3)
        ax0.hlines(inds, quartile1, quartile3, color='k', linestyle='-', lw=8)
        ax0.hlines(inds, whiskers_min, whiskers_max, color='k', linestyle='-', lw=1)
        ax0.set_xlabel('Pixels normalized')
        ax0.set_yticks([])
        ax0.set_yticklabels([])

        ax1 = fig.add_subplot(gs[1, :])
        if unit.lower() == 'seconds':
            ax1.set_xlabel('Time [seconds]')
        else:
            ax1.set_xlabel('Time [samples]')
            fps = 1
        ax1.set_ylabel('Pixels normalized')
        ax1.bar(np.arange(len(qom)-1)/fps, qom[1:]/(max(qom[1:])))

    # All motion data
    if motion_analysis.lower() == 'all':
        ax0 = fig.add_subplot(gs[0, 0])
        ax0.scatter(com[:, 0]/width, com[:, 1]/height, s=2)
        ax0.set_xlim((0, 1))
        ax0.set_ylim((0, 1))
        ax0.set_xlabel('Pixels normalized')
        ax0.set_ylabel('Pixels normalized')
        ax0.set_title('Centroid of motion (CoM)')

        ax0 = fig.add_subplot(gs[0, 1])
        ax0.scatter(aom[:, 0], aom[:, 1], c='C0', s=2)
        ax0.scatter(aom[:, 2], aom[:, 3], c='C0', s=2)
        ax0.set_xlim((0, 1))
        ax0.set_ylim((0, 1))
        ax0.set_xlabel('Pixels normalized')
        ax0.set_ylabel('Pixels normalized')
        ax0.set_title('Area of motion (AoM)')

        ax1 = fig.add_subplot(gs[1, :])
        ax1.set_title('Quantity of motion (QoM)')
        ax1.violinplot([qom[1:]/(max(qom[1:]))], showmeans=False, showmedians=True, showextrema=True, vert=False)
        quartile1, medians, quartile3 = np.percentile([qom[1:]/(max(qom[1:]))], [25, 50, 75], axis=1)
        whiskers = np.array([adjacent_values(sorted_array, q1, q3) for sorted_array, q1, q3 in zip([qom[1:]/(max(qom[1:]))], quartile1, quartile3)])
        whiskers_min, whiskers_max = whiskers[:, 0], whiskers[:, 1]
        inds = np.arange(1, len(medians) + 1)
        ax1.scatter(medians, inds, marker='o', color='white', s=60, zorder=3)
        ax1.hlines(inds, quartile1, quartile3, color='k', linestyle='-', lw=8)
        ax1.hlines(inds, whiskers_min, whiskers_max, color='k', linestyle='-', lw=1)
        ax1.set_xlabel('Pixels normalized')
        ax1.set_yticks([])
        ax1.set_yticklabels([])

        ax2 = fig.add_subplot(gs[2, :])
        if unit.lower() == 'seconds':
            ax2.set_xlabel('Time [seconds]')
        else:
            ax2.set_xlabel('Time [samples]')
            fps = 1
        ax2.set_ylabel('Pixels normalized')
        ax2.bar(np.arange(len(qom)-1)/fps, qom[1:]/(max(qom[1:])))

    # Plot audio descriptors
    if audio_descriptors:

        freq_ticks = [elem*100 for elem in range(10)]
        freq_ticks = [250]
        freq = 500
        while freq < descriptors['sr']/2:
            freq_ticks.append(freq)
            freq *= 1.5

        freq_ticks = [round(elem, -1) for elem in freq_ticks]
        freq_ticks_labels = [str(round(elem/1000, 1)) + 'k' if elem > 1000 else int(round(elem)) for elem in freq_ticks]

        times = librosa.times_like(descriptors['cent'], sr=descriptors['sr'], n_fft=2048, hop_length=descriptors['hop_size'])

        if unit.lower() == 'samples':
            times = times*descriptors['sr']

        if motion_analysis.lower() == 'all':
            ax3 = fig.add_subplot(gs[3, :], sharex=ax2)
        elif motion_analysis.lower() == 'qom':
            ax3 = fig.add_subplot(gs[2, :])
        else:
            ax3 = fig.add_subplot(gs[1, :])

        ax3.set_title('Audio descriptors')
        ax3.semilogy(times, descriptors['rms'][0], label='RMS Energy')
        ax3.legend(loc='upper right')

        if motion_analysis.lower() == 'all':
            ax4 = fig.add_subplot(gs[4, :], sharex=ax2)
        elif motion_analysis.lower() == 'qom':
            ax4 = fig.add_subplot(gs[3, :])
        else:
            ax4 = fig.add_subplot(gs[2, :])

        ax4.plot(times, descriptors['flatness'].T, label='Flatness', color='y')
        ax4.legend(loc='upper right')

        if motion_analysis.lower() == 'all':
            ax5 = fig.add_subplot(gs[5, :], sharex=ax2)
        elif motion_analysis.lower() == 'qom':
            ax5 = fig.add_subplot(gs[4, :])
        else:
            ax5 = fig.add_subplot(gs[3, :])

        ax5.set_ylabel('Frequency [Hz]')
        ax5.fill_between(times, descriptors['cent'][0] - descriptors['spec_bw'][0], descriptors['cent'][0] + descriptors['spec_bw'][0], alpha=0.5, label='Centroid +- bandwidth')
        ax5.plot(times, descriptors['cent'].T, label='Centroid', color='y')
        ax5.plot(times, descriptors['rolloff'][0], label='Roll-off frequency (0.99)')
        ax5.plot(times, descriptors['rolloff_min'][0], color='r',label='Roll-off frequency (0.01)')
        ax5.legend(loc='upper right')

        if unit.lower() == 'seconds':
            ax5.set_xlabel('Time [seconds]')
        else:
            ax5.set_xlabel('Time [samples]')

    fig.tight_layout()

    if target_name_plot is None:
        target_name_plot = of + '_motionplot.png'
    else:
        # enforce png
        target_name_plot = os.path.splitext(target_name_plot)[0] + '.png'
    if not overwrite:
        target_name_plot = generate_outfilename(target_name_plot)

    plt.savefig(target_name_plot, format='png', transparent=False)
    plt.close()

    return target_name_plot

show

show(filename=None, key=None, mode='windowed', window_width=640, window_height=480, window_title=None, **ipython_kwargs)

General method to show an image or video file either in a window, or inline in a jupyter notebook.

Parameters:

Name	Type	Description	Default
filename	str	If given, mg_show will show this file instead of what it inherits from its parent object. Defaults to None.	None
key	str	If given, mg_show will search for file names corresponding to certain processes you have previously rendered on your source. It is meant to be a shortcut, so you don't have to remember the exact name (and path) of eg. a motion video corresponding to your source in your MgVideo, but you rather just use MgVideo('path/to/vid.mp4').show(key='motion'). Accepted values are 'horizontal'/'vertical' (motiongram or videogram; aliases 'mgh'/'vgh' for horizontal, 'mgv'/'vgv' for vertical, and the legacy 'mgx'/'mgy'/'vgx'/'vgy' literal x/y files), 'ssm', 'blend', 'plot', 'motion', 'history', 'motionhistory', 'sparse', 'dense', 'pose', 'warp', 'blur', and 'subtract'. Defaults to None.	None
mode	str	Whether to show things in a separate window or inline in the jupyter notebook. Accepted values are 'windowed' and 'notebook'. Defaults to 'windowed'.	'windowed'
window_width	int	The width of the window. Defaults to 640.	640
window_height	int	The height of the window. Defaults to 480.	480
window_title	str	The title of the window. If None, the title of the window will be the file name. Defaults to None.	None
ipython_kwargs	dict	Additional arguments for IPython.display.Image or IPython.display.Video. Defaults to None.	{}

Source code in musicalgestures/_show.py

def mg_show(self, filename: str | None = None, key: str | None = None, mode: str = 'windowed', window_width: int = 640, window_height: int = 480, window_title: str | None = None, **ipython_kwargs):
    # def mg_show(self, filename=None, mode='windowed', window_width=640, window_height=480, window_title=None):
    """
    General method to show an image or video file either in a window, or inline in a jupyter notebook.

    Args:
        filename (str, optional): If given, `mg_show` will show this file instead of what it inherits from its parent object. Defaults to None.
        key (str, optional): If given, `mg_show` will search for file names corresponding to certain processes you have previously rendered on your source. It is meant to be a shortcut, so you don't have to remember the exact name (and path) of eg. a motion video corresponding to your source in your MgVideo, but you rather just use `MgVideo('path/to/vid.mp4').show(key='motion')`. Accepted values are 'horizontal'/'vertical' (motiongram or videogram; aliases 'mgh'/'vgh' for horizontal, 'mgv'/'vgv' for vertical, and the legacy 'mgx'/'mgy'/'vgx'/'vgy' literal x/y files), 'ssm', 'blend', 'plot', 'motion', 'history', 'motionhistory', 'sparse', 'dense', 'pose', 'warp', 'blur', and 'subtract'. Defaults to None.
        mode (str, optional): Whether to show things in a separate window or inline in the jupyter notebook. Accepted values are 'windowed' and 'notebook'. Defaults to 'windowed'.
        window_width (int, optional): The width of the window. Defaults to 640.
        window_height (int, optional): The height of the window. Defaults to 480.
        window_title (str, optional): The title of the window. If None, the title of the window will be the file name. Defaults to None.
        ipython_kwargs (dict, optional): Additional arguments for IPython.display.Image or IPython.display.Video. Defaults to None.
    """
    # Lazy import: keeps IPython out of `import musicalgestures` startup.
    from IPython.display import Image, display, HTML, Video

    def show(file, width=640, height=480, mode='windowed', title='Untitled', parent=None, **ipython_kwargs):
        """
        Helper function which actually does the "showing".

        Args:
            file (str): Path to the file.
            width (int, optional): The width of the window. Defaults to 640.
            height (int, optional): The height of the window. Defaults to 480.
            mode (str, optional): 'windowed' will use ffplay (in a separate window), while 'notebook' will use Image or Video from IPython.display. Defaults to 'windowed'.
            title (str, optional): The title of the window. Defaults to 'Untitled'.
            ipython_kwargs (dict, optional): Additional arguments for IPython.display.Image or IPython.display.Video. Defaults to None.
        """

        # Check's if the environment is a Google Colab document
        if musicalgestures._utils.in_colab():
            mode = 'notebook'
        elif musicalgestures._utils.in_ipynb():
            mode = 'notebook'

        if mode.lower() == 'windowed':
            # from musicalgestures._utils import wrap_str
            # cmd = f'ffplay {wrap_str(file)} -window_title {wrap_str(title)} -x {width} -y {height}'

            video_to_display = os.path.realpath(file)
            cmd = ' '.join(map(str, ['ffplay', video_to_display, '-window_title', title, '-x', width, '-y', height]))
            show_in_new_process(cmd)      

        elif mode.lower() == 'notebook':
            video_formats = ['.avi', '.mp4', '.mov', '.mkv', '.mpg', '.mpeg', '.webm', '.ogg', '.ts', '.wmv', '.3gp', '.lrv', '.insv', '.360', '.glv']
            image_formats = ['.jpg', '.png', '.jpeg', '.tiff', '.gif', '.bmp']

            of, file_extension = os.path.splitext(file)
            file_extension = file_extension.lower()

            if file_extension in video_formats:
                file_type = 'video'
            elif file_extension in image_formats:
                file_type = 'image'

            if file_type == 'image':
                display(Image(file))    
            elif file_type == 'video':
                if file_extension not in ['.mp4', '.webm', '.ogg']:
                    keys = parent.__dict__.keys()

                    if "as_mp4" not in keys:
                        print('Only mp4, webm and ogg videos are supported in notebook mode.')
                        video_to_display = musicalgestures._utils.convert_to_mp4(file)
                        # register converted video as_mp4 for parent MgVideo
                        parent.as_mp4 = musicalgestures.MgVideo(video_to_display)
                    else:
                        video_to_display = parent.as_mp4.filename
                else:
                    video_to_display = file

                # check width and height of video, if they are bigger than "appropriate", limit their dimensions
                video_width, video_height = musicalgestures._utils.get_widthheight(video_to_display)
                video_width = video_width if video_width <= width else width
                video_height = video_height if video_height <= height else height

                # if the video is at the same folder as the notebook, we need to use relative path
                # and if it is somewhere else, we need to embed it to make it work (neither absolute nor relative paths seem to work without embedding)
                cwd = os.getcwd().replace('\\', '/')
                file_dir = os.path.dirname(video_to_display).replace('\\', '/')

                def colab_display(video_to_display, video_width, video_height):
                  video_file = open(video_to_display, "r+b").read()
                  video_url = f"data:video/mp4;base64,{b64encode(video_file).decode()}"
                  return HTML(f"""<video width={video_width} height={video_height} controls><source src="{video_url}"></video>""")

                if file_dir == cwd:
                    try:
                        video_to_display = os.path.relpath(video_to_display, os.getcwd()).replace('\\', '/')
                        if musicalgestures._utils.in_colab():
                            display(colab_display(video_to_display, video_width, video_height))
                        else:
                            display(Video(video_to_display,width=video_width, height=video_height, **ipython_kwargs))
                    except ValueError:
                        video_to_display = os.path.abspath(video_to_display, os.getcwd()).replace('\\', '/')
                        if musicalgestures._utils.in_colab():
                            display(colab_display(video_to_display, video_width, video_height))
                        else:
                            display(Video(video_to_display, width=video_width, height=video_height, **ipython_kwargs))
                else:
                    try:
                        video_to_display = os.path.relpath(video_to_display, os.getcwd()).replace('\\', '/')
                        if musicalgestures._utils.in_colab():
                            display(colab_display(video_to_display, video_width, video_height))
                        else:
                            display(Video(video_to_display, width=video_width, height=video_height, **ipython_kwargs))
                    except ValueError:
                        video_to_display = os.path.abspath(video_to_display, os.getcwd()).replace('\\', '/')
                        if musicalgestures._utils.in_colab():
                            display(colab_display(video_to_display, video_width, video_height))
                        else:
                            display(Video(video_to_display, width=video_width,height=video_height, **ipython_kwargs))

        else:
            print(f'Unrecognized mode: "{mode}". Try "windowed" or "notebook".')

    if window_title is None:
        window_title = self.filename

    if filename is None:
        keys = self.__dict__.keys()
        if key is None:
            filename = self.filename
            show(file=filename, width=window_width,
                 height=window_height, mode=mode, title=window_title, parent=self, **ipython_kwargs)

        elif key.lower() in ('horizontal', 'vertical', 'mgh', 'mgv', 'vgh', 'vgv',
                             'mgx', 'mgy', 'vgx', 'vgy'):
            # Orientation keys for motiongrams/videograms. Horizontal movement is captured by
            # the y-axis collapse (motiongram_y / videogram_y), vertical by the x-axis collapse
            # (motiongram_x / videogram_x). Canonical aliases: mgh/vgh (horizontal),
            # mgv/vgv (vertical); legacy mgx/mgy/vgx/vgy map to the literal x/y files.
            k = key.lower()
            horizontal_keys = ('horizontal', 'mgh', 'vgh', 'mgy', 'vgy')
            axis = 'y' if k in horizontal_keys else 'x'
            label = 'Horizontal' if k in horizontal_keys else 'Vertical'
            if k in ('mgh', 'mgv', 'mgx', 'mgy'):
                kinds = ('motiongram',)
            elif k in ('vgh', 'vgv', 'vgx', 'vgy'):
                kinds = ('videogram',)
            else:
                kinds = ('motiongram', 'videogram')
            target = None
            for kind in kinds:
                if f"{kind}_{axis}" in keys:
                    target = (kind, getattr(self, f"{kind}_{axis}").filename)
                    break
            if target is None:
                raise FileNotFoundError(
                    f"There is no known {label.lower()} {' or '.join(kinds)} for this file. "
                    "Run motiongrams() or videograms() first.")
            kind, filename = target
            show(file=filename, width=window_width, height=window_height, mode=mode,
                 title=f'{label} {kind.capitalize()} | {filename}', parent=self, **ipython_kwargs)

        elif key.lower() == 'ssm':
            if "ssm_fig" in keys:
                filename = self.ssm_fig.image
                if len(filename) == 2:
                    show(file=filename[0], width=window_width, height=window_height, mode=mode, title=f'Horizontal SSM | {filename}', parent=self, **ipython_kwargs)
                    show(file=filename[1], width=window_width, height=window_height, mode=mode, title=f'Vertical SSM | {filename}', parent=self, **ipython_kwargs)
                else:    
                    show(file=filename, width=window_width, height=window_height, mode=mode, title=f'Self-Similarity Matrix | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known self-smilarity matrix for this file.")

        elif key.lower() == 'blend':
            if "blend_image" in keys:
                filename = self.blend_image.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Blended Image | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known blended image for this file.")

        elif key.lower() == 'plot':
            # filename = self.of + '_motion_com_qom.png'
            if "motion_plot" in keys:
                filename = self.motion_plot.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Centroid and Quantity of Motion | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known motion plot for this file.")

        elif key.lower() == 'motion':
            if "motion_video" in keys:
                filename = self.motion_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Motion Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known motion video for this file.")

        elif key.lower() == 'history':
            if "history_video" in keys:
                filename = self.history_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'History Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known history video for this file.")

        elif key.lower() == 'motionhistory':
            if "motion_video" in keys:
                motion_video_keys = self.motion_video.__dict__.keys()
                if "history_video" in motion_video_keys:
                    filename = self.motion_video.history_video.filename
                    show(file=filename, width=window_width,
                         height=window_height, mode=mode, title=f'Motion History Video | {filename}', parent=self, **ipython_kwargs)
                else:
                    raise FileNotFoundError(
                        "There is no known motion history video for this file.")
            else:
                raise FileNotFoundError(
                    "There is no known motion video for this file.")

        elif key.lower() == 'sparse':
            if "flow_sparse_video" in keys:
                filename = self.flow_sparse_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Sparse Optical Flow Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known sparse optial flow video for this file.")

        elif key.lower() == 'dense':
            if "flow_dense_video" in keys:
                filename = self.flow_dense_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Dense Optical Flow Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known dense optial flow video for this file.")

        elif key.lower() == 'pose':
            if "pose_video" in keys:
                filename = self.pose_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Pose Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known pose video for this file.")

        elif key.lower() == 'warp':
            if "warp_video" in keys:
                filename = self.warp_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Warp Audiovisual Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known warp audiovisual beats video for this file.")

        elif key.lower() == 'blur':
            if "blur_faces_video" in keys:
                filename = self.blur_faces_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Blur Faces Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError(
                    "There is no known blur faces video for this file.")

        elif key.lower() == 'subtract':
            if "subtract_video" in keys:
                filename = self.subtract_video.filename
                show(file=filename, width=window_width,
                     height=window_height, mode=mode, title=f'Background Subtraction Video | {filename}', parent=self, **ipython_kwargs)
            else:
                raise FileNotFoundError("There is no known subtract video for this file.")

        else:
            print("Unknown shorthand.\n",
                  "For images, try 'horizontal', 'vertical', 'ssm', 'blend' or 'plot'.\n",
                  "For videos try 'motion', 'history', 'motionhistory', 'sparse', 'dense', 'pose', 'warp', 'blur' or 'subtract'.")

    else:
        show(file=filename, width=window_width,
             height=window_height, mode=mode, title=window_title, parent=self, **ipython_kwargs)
    # show(file=filename, width=window_width, height=window_height, mode=mode, title=window_title)

    return self

info

info(type=None, autoshow=True, overwrite=True)

Returns info about video/audio/format file using ffprobe.

Parameters:

Name	Type	Description	Default
type	str	Type of information to retrieve. Possible choices are 'summary', 'audio', 'video', 'format' or 'frame'. Defaults to None (which gives info about video, audio and format). - 'summary': prints a human-readable table of key video properties (resolution, fps, frame count, duration, color mode, audio) and returns a dict. - 'audio' / 'video' / 'format': returns the matching ffprobe stream as a pandas DataFrame row. - 'frame': renders a bar chart of I/P/B frame sizes and returns a DataFrame. - None: returns a DataFrame with all ffprobe stream and format metadata.	None
autoshow	bool	Whether to show the I/P/B frames figure automatically. Defaults to True. NB: The type argument needs to be set to 'frame'.	True
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filename to avoid overwriting. Defaults to True.	True

Returns:

Type	Description
	dict or pandas.DataFrame: dict when type='summary', DataFrame otherwise.

Source code in musicalgestures/_info.py

def mg_info(self, type=None, autoshow=True, overwrite=True):
    """
    Returns info about video/audio/format file using ffprobe.

    Args:
        type (str, optional): Type of information to retrieve. Possible choices are 'summary', 'audio', 'video', 'format' or 'frame'. Defaults to None (which gives info about video, audio and format).
            - 'summary': prints a human-readable table of key video properties (resolution, fps, frame count, duration, color mode, audio) and returns a dict.
            - 'audio' / 'video' / 'format': returns the matching ffprobe stream as a pandas DataFrame row.
            - 'frame': renders a bar chart of I/P/B frame sizes and returns a DataFrame.
            - None: returns a DataFrame with all ffprobe stream and format metadata.
        autoshow (bool, optional): Whether to show the I/P/B frames figure automatically. Defaults to True. NB: The type argument needs to be set to 'frame'.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filename to avoid overwriting. Defaults to True.

    Returns:
        dict or pandas.DataFrame: dict when type='summary', DataFrame otherwise.
    """

    if type == 'summary':
        framecount = get_framecount(self.filename)

        # NB: for an MgVideo `self.length` is the frame *count*, not seconds
        # (for an MgAudio it is the duration in seconds and there is no fps).
        fps = getattr(self, 'fps', None)
        duration_secs = self.length / fps if fps else float(self.length)

        h = int(duration_secs // 3600)
        m = int((duration_secs % 3600) // 60)
        s = duration_secs % 60
        duration_str = f"{h}:{m:02d}:{s:05.2f}" if h else f"{m}:{s:05.2f}"

        filesize = os.path.getsize(self.filename)
        if filesize >= 1_000_000:
            size_str = f"{filesize / 1_000_000:.1f} MB"
        elif filesize >= 1_000:
            size_str = f"{filesize / 1_000:.1f} KB"
        else:
            size_str = f"{filesize} B"

        # Query codec/profile details from ffprobe
        v = _probe_stream(self.filename, 'v')
        a = _probe_stream(self.filename, 'a')

        video_codec = v.get('codec_name')
        video_profile = v.get('profile')
        pix_fmt = v.get('pix_fmt')
        color_space = v.get('color_space')
        color_profile = ', '.join(x for x in (pix_fmt, color_space) if x and x != 'unknown') or None

        audio_codec = a.get('codec_name')
        audio_sr = a.get('sample_rate')
        audio_br = a.get('bit_rate')
        audio_sr_str = f"{int(audio_sr):,} Hz" if audio_sr and audio_sr.isdigit() else None
        audio_br_str = f"{int(audio_br) // 1000} kbps" if audio_br and audio_br.isdigit() else None

        info_dict = {
            'filename':       os.path.basename(self.filename),
            'width':          self.width,
            'height':         self.height,
            'fps':            self.fps,
            'frames':         framecount,
            'duration':       round(duration_secs, 3),
            'color':          self.color,
            'video_codec':    video_codec,
            'video_profile':  video_profile,
            'pixel_format':   pix_fmt,
            'color_space':    color_space,
            'has_audio':      bool(self.has_audio),
            'audio_codec':    audio_codec,
            'audio_sample_rate': int(audio_sr) if audio_sr and audio_sr.isdigit() else None,
            'audio_bit_rate': int(audio_br) if audio_br and audio_br.isdigit() else None,
            'filesize':       filesize,
        }

        col = 14
        print(f"{'File:':<{col}} {os.path.basename(self.filename)}")
        print(f"{'Resolution:':<{col}} {self.width} × {self.height} px")
        print(f"{'Frames:':<{col}} {framecount}  @  {self.fps:g} fps")
        print(f"{'Duration:':<{col}} {duration_str}  ({duration_secs:.3f} s)")
        print(f"{'Color:':<{col}} {'color' if self.color else 'grayscale'}")
        codec_str = video_codec or 'unknown'
        if video_profile:
            codec_str += f" ({video_profile})"
        print(f"{'Video codec:':<{col}} {codec_str}")
        if color_profile:
            print(f"{'Color profile:':<{col}} {color_profile}")
        if self.has_audio:
            audio_str = audio_codec or 'unknown'
            extras = ', '.join(x for x in (audio_sr_str, audio_br_str) if x)
            if extras:
                audio_str += f" ({extras})"
            print(f"{'Audio:':<{col}} {audio_str}")
        else:
            print(f"{'Audio:':<{col}} no")
        print(f"{'File size:':<{col}} {size_str}")

        return info_dict

    # Get streams and format information (https://ffmpeg.org/ffprobe.html)
    cmd = ["ffprobe", "-hide_banner", "-loglevel", "quiet", "-show_streams", "-show_format", self.filename]
    if type == 'frame':
        if self.fex != '.mp4':
            # Convert video file to mp4 
            self.filename = convert_to_mp4(self.of + self.fex, overwrite=overwrite)
            self.of, self.fex = os.path.splitext(self.filename)
        cmd = ["ffprobe", "-hide_banner", "-loglevel", "quiet", "-v", "error", "-select_streams", "v:0", "-show_entries", "frame=pkt_size, pict_type", self.filename]

    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
    try:
        out, _ = process.communicate(timeout=10)
        splitted = out.split('\n')
    except subprocess.TimeoutExpired:
        process.kill()
    out, err = process.communicate()
    splitted = out.split('\n')

    frame = []

    # Retrieve information and export it in a dictionary
    if type == 'frame':
        current_frame = {}
        for line in [i for i in splitted if i not in ('[SIDE_DATA]', '[/SIDE_DATA]', '')]:
            if line == '[/FRAME]':
                frame.append(current_frame)
                current_frame = {}
            elif line != '[FRAME]':
                pair = line.split('=')
                current_frame[pair[0]] = pair[1]
            else:
                pass

        ipb_frames = {
                      'frame index': range(len(frame)),
                      'size (bytes)': [int(f['pkt_size']) for f in frame],
                      'type': [f['pict_type'] for f in frame]
                      }

        df = pd.DataFrame.from_dict(ipb_frames)

        if not autoshow:
            return df

        fig, ax = plt.subplots(figsize=(12,4), dpi=300)
        fig.patch.set_facecolor('white') # make sure background is white
        fig.patch.set_alpha(1)

        for i, (label, series) in enumerate(df.groupby('type')):
            plot_frames(series, label, index=i)

        # Get handles and labels
        handles, labels = plt.gca().get_legend_handles_labels()
        order = [1,2,0] # specify order of items in legend
        # Add legend to plot
        ax.legend([handles[idx] for idx in order],[labels[idx] for idx in order])
        ax.set_xlabel('Frame index')
        ax.set_ylabel('Size (bytes)')
        fig.tight_layout()

        # Save and close so display goes through the returned MgImage's show()
        target_png = self.of + '_frames.png'
        if not overwrite:
            target_png = generate_outfilename(target_png)
        fig.savefig(target_png, facecolor='white')
        plt.close(fig)
        return MgImage(target_png)

    else:
        for i, info in enumerate(splitted):
            if info == "[STREAM]" or info == "[SIDE_DATA]" or info == "[FORMAT]":        
                frame.append(dict())
                i +=1
            elif info == "[/STREAM]" or info == "[/SIDE_DATA]" or info == "[/FORMAT]" or info == "":
                i +=1
            else:
                try:
                    key, value = splitted[i].split('=')
                    frame[-1][key] = value
                except ValueError:
                    key = splitted[i]
                    frame[-1][key] = ''

        if len(frame) > 3: 
            # Merge video stream with side data dictionary
            frame[0] = {**frame[0], **frame[1]}
            frame.pop(1)

        # Create a pandas dataframe
        df = pd.DataFrame.from_dict(frame)

        df.insert(0, 'codec_type', df.pop('codec_type')) # move codec type column
        df.pop('index') # remove index column
        df = df[df.codec_type.notna()] # remove rows with nan values in codec_type column

        if type is not None:
            return df[df.codec_type == type]
        else:
            return df

history

history(filename=None, history_length=10, weights=1, normalize=False, norm_strength=1, norm_smooth=0, target_name=None, overwrite=True)

This function creates a video where each frame is the average of the N previous frames, where n is determined by history_length. The history frames are summed up and normalized, and added to the current frame to show the history. Uses ffmpeg.

Parameters:

Name	Type	Description	Default
filename	str	Path to the input video file. If None, the video file of the MgVideo is used. Defaults to None.	None
history_length	int	Number of frames to be saved in the history tail. Defaults to 10.	10
weights	int / float / list / str	Defines the weight or weights applied to the frames in the history tail. If given as list the first element in the list will correspond to the weight of the newest frame in the tail. If given as a str - like "3 1.2 1" - it will be automatically converted to a list - like [3, 1.2, 1]. Defaults to 1.	1
normalize	bool	If True, the history video will be normalized. This can be useful when processing motion (frame difference) videos. Defaults to False.	False
norm_strength	int / float	Defines the strength of the normalization where 1 represents full strength. Defaults to 1.	1
norm_smooth	int	Defines the number of previous frames to use for temporal smoothing. The input range of each channel is smoothed using a rolling average over the current frame and the norm_smooth previous frames. Defaults to 0.	0
target_name	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_history" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo		A new MgVideo pointing to the output video file.

Source code in musicalgestures/_history.py

def history_ffmpeg(self, filename: str | None = None, history_length: int = 10, weights: int | float | list | str = 1, normalize: bool = False, norm_strength: int | float = 1, norm_smooth: int = 0, target_name: str | None = None, overwrite: bool = True):
    """
    This function  creates a video where each frame is the average of the N previous frames, where n is determined by `history_length`. The history frames are summed up and normalized, and added to the current frame to show the history. Uses ffmpeg.

    Args:
        filename (str, optional): Path to the input video file. If None, the video file of the MgVideo is used. Defaults to None.
        history_length (int, optional): Number of frames to be saved in the history tail. Defaults to 10.
        weights (int/float/list/str, optional): Defines the weight or weights applied to the frames in the history tail. If given as list the first element in the list will correspond to the weight of the newest frame in the tail. If given as a str - like "3 1.2 1" - it will be automatically converted to a list - like [3, 1.2, 1]. Defaults to 1.
        normalize (bool, optional): If True, the history video will be normalized. This can be useful when processing motion (frame difference) videos. Defaults to False.
        norm_strength (int/float, optional): Defines the strength of the normalization where 1 represents full strength. Defaults to 1.
        norm_smooth (int, optional): Defines the number of previous frames to use for temporal smoothing. The input range of each channel is smoothed using a rolling average over the current frame and the `norm_smooth` previous frames. Defaults to 0.
        target_name (str, optional): Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_history" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A new MgVideo pointing to the output video file.
    """

    if filename is None:
        filename = self.filename

    of, fex = os.path.splitext(filename)

    if type(weights) in [int, float]:
        weights_map = np.ones(history_length)
        weights_map[-1] = weights
        str_weights = ' '.join([str(weight) for weight in weights_map])
    elif type(weights) == list:
        typecheck_list = [type(item) in [int, float] for item in weights]
        if False in typecheck_list:
            raise MgInputError(
                'Found wrong type(s) in the list of weights. Use ints and floats.')
        elif len(weights) > history_length:
            raise MgInputError(
                'history_length must be greater than or equal to the number of weights specified in weights.')
        else:
            weights_map = np.ones(history_length - len(weights))
            weights.reverse()
            weights_map = list(weights_map)
            weights_map += weights
            str_weights = ' '.join([str(weight) for weight in weights_map])
    elif type(weights) == str:
        weights_as_list = weights.split()
        try:
            weights_as_list = [float(item) for item in weights_as_list]
        except ValueError:
            raise MgInputError(
                'Found wrong type(s) in the list of weights. Use ints and floats.')
        if len(weights_as_list) > history_length:
            raise MgInputError(
                'history_length must be greater than or equal to the number of weights specified in weights.')
        else:
            weights_map = np.ones(history_length - len(weights_as_list))
            weights_as_list.reverse()
            weights_map = list(weights_map)
            weights_map += weights_as_list
            str_weights = ' '.join([str(weight) for weight in weights_map])
    else:
        raise MgInputError(
            'Wrong type used for weights. Use int, float, str, or list.')

    if type(normalize) != bool:
        raise MgInputError(
            'Wrong type used for normalize. Use only bool.')

    if normalize:
        if type(norm_strength) not in [int, float]:
            raise MgInputError(
                'Wrong type used for norm_strength. Use int or float.')
        if type(norm_smooth) != int:
            raise MgInputError(
                'Wrong type used for norm_smooth. Use only int.')

    target_name = resolve_filename(of, '_history' + fex, target_name, overwrite)

    if normalize:
        if norm_smooth != 0:
            cmd = ['ffmpeg', '-y', '-i', filename, '-filter_complex',
                   f'tmix=frames={history_length}:weights={str_weights},normalize=independence=0:strength={norm_strength}:smoothing={norm_smooth}', '-q:v', '3', '-c:a', 'copy', target_name]
        else:
            cmd = ['ffmpeg', '-y', '-i', filename, '-filter_complex',
                   f'tmix=frames={history_length}:weights={str_weights},normalize=independence=0:strength={norm_strength}', '-q:v', '3', '-c:a', 'copy', target_name]
    else:
        cmd = ['ffmpeg', '-y', '-i', filename, '-vf',
               f'tmix=frames={history_length}:weights={str_weights}', '-q:v', '3', '-c:a', 'copy', target_name]

    ffmpeg_cmd(cmd, get_length(filename), pb_prefix='Rendering history video:')

    # save the result as the history_video for parent MgVideo
    self.history_video = musicalgestures.MgVideo(
        target_name, color=self.color, returned_by_process=True)

    return self.history_video

history_cv2

history_cv2(filename=None, history_length=10, weights=1, convert=True, target_name=None, overwrite=True)

This function creates a video where each frame is the average of the N previous frames, where n is determined by history_length. The history frames are summed up and normalized, and added to the current frame to show the history. Uses cv2.

Parameters:

Name	Type	Description	Default
filename	str	Path to the input video file. If None, the video file of the MgVideo is used. Defaults to None.	None
history_length	int	Number of frames to be saved in the history tail. Defaults to 10.	10
weights	int / float / list	Defines the weight or weights applied to the frames in the history tail. If given as list the first element in the list will correspond to the weight of the newest frame in the tail. Defaults to 1.	1
convert	bool	If True (default), non-AVI input is first converted to an all-intra MJPEG .avi (cached as self.as_avi) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.	True
target_name	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_history" should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo		A new MgVideo pointing to the output video file.

Source code in musicalgestures/_history.py

def history_cv2(self, filename: str | None = None, history_length: int = 10, weights: int | float | list = 1, convert: bool = True, target_name: str | None = None, overwrite: bool = True):
    """
    This function  creates a video where each frame is the average of the N previous frames, where n is determined by `history_length`. The history frames are summed up and normalized, and added to the current frame to show the history. Uses cv2.

    Args:
        filename (str, optional): Path to the input video file. If None, the video file of the MgVideo is used. Defaults to None.
        history_length (int, optional): Number of frames to be saved in the history tail. Defaults to 10.
        weights (int/float/list, optional): Defines the weight or weights applied to the frames in the history tail. If given as list the first element in the list will correspond to the weight of the newest frame in the tail. Defaults to 1.
        convert (bool, optional): If True (default), non-AVI input is first converted to an all-intra MJPEG `.avi` (cached as `self.as_avi`) for frame-accurate decoding. Set to False to read the source file directly. Defaults to True.
        target_name (str, optional): Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_history" should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: A new MgVideo pointing to the output video file.
    """

    if filename is None:
        filename = self.filename

    of, fex = os.path.splitext(filename)

    if convert and fex != '.avi':
        # first check if there already is a converted version, if not create one and register it to the parent self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex
        filename = of + fex

    video = cv2.VideoCapture(filename)
    ret, frame = video.read()
    fourcc = cv2.VideoWriter_fourcc(*'MJPG')

    fps = int(video.get(cv2.CAP_PROP_FPS))
    width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
    length = int(video.get(cv2.CAP_PROP_FRAME_COUNT))

    pb = MgProgressbar(total=length, prefix='Rendering history video:')

    target_name = resolve_filename(of, '_history' + fex, target_name, overwrite)

    out = cv2.VideoWriter(target_name, fourcc, fps, (width, height))

    ii = 0
    history = []
    weights_map = [1 for weight in range(history_length+1)]

    if type(weights) in [int, float]:
        offset = weights - 1
        weights_map[0] = weights
    elif type(weights) == list:
        offset = sum([weight - 1 for weight in weights])
        for ind, weight in enumerate(weights):
            if ind > history_length:
                break
            weights_map[ind] = weight

    denominator = history_length + 1 + offset

    while(video.isOpened()):
        ret, frame = video.read()
        if ret == True:
            if self.color == False:
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            frame = (np.array(frame)).astype(np.float32)

            if len(history) > 0:
                #history_total = frame/(len(history)+1)
                history_total = frame * weights_map[0] / denominator
                # history_total = frame
            else:
                history_total = frame

            for ind, newframe in enumerate(history):
                #history_total += newframe/(len(history)+1)
                history_total += newframe * weights_map[ind+1] / denominator
            # or however long history you would like
            if len(history) >= history_length:
                history.pop(0)  # pop first frame
            history.append(frame)
            # 0.5 to not overload it poor thing
            total = history_total.astype(np.uint64)

            if self.color == False:
                total = cv2.cvtColor(total.astype(
                    np.uint8), cv2.COLOR_GRAY2BGR)
                out.write(total)
            else:
                out.write(total.astype(np.uint8))

        else:
            pb.progress(length)
            break

        pb.progress(ii)
        ii += 1

    out.release()

    destination_video = target_name

    if self.has_audio:
        source_audio = extract_wav(self.of + self.fex)
        embed_audio_in_video(source_audio, destination_video)
        os.remove(source_audio)

    self.history_video = musicalgestures.MgVideo(
        destination_video, color=self.color, returned_by_process=True)

    # return musicalgestures.MgVideo(destination_video, color=self.color, returned_by_process=True)
    return self.history_video

blend

blend(filename=None, mode='all_mode', component_mode='average', target_name=None, overwrite=True)

Finds and saves a blended image of an input video file using FFmpeg. The FFmpeg tblend (time blend) filter takes two consecutive frames from one single stream, and outputs the result obtained by blending the new frame on top of the old frame.

Parameters:

Name	Type	Description	Default
filename	str	Path to the input video file. If None, the video file of the MgObject is used. Defaults to None.	None
mode	str	Set blend mode for specific pixel component or all pixel components. Accepted options are 'c0_mode', 'c1_mode', c2_mode', 'c3_mode' and 'all_mode'. Defaults to 'all_mode'.	'all_mode'
component_mode	str	Component mode of the FFmpeg tblend. Available values for component modes can be accessed here: https://ffmpeg.org/ffmpeg-filters.html#blend-1. Defaults to 'average'.	'average'
target_name	str	The name of the output video. Defaults to None (which assumes that the input filename with the component mode suffix should be used).	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	MgImage	A new MgImage pointing to the output image file.

Source code in musicalgestures/_blend.py

def mg_blend_image(self, filename=None, mode='all_mode', component_mode='average', target_name=None, overwrite=True) -> "MgImage":
    """
    Finds and saves a blended image of an input video file using FFmpeg. 
    The FFmpeg tblend (time blend) filter takes two consecutive frames from one single stream, and outputs the result obtained by blending the new frame on top of the old frame.

    Args:
        filename (str, optional): Path to the input video file. If None, the video file of the MgObject is used. Defaults to None.
        mode (str, optional): Set blend mode for specific pixel component or all pixel components. Accepted options are 'c0_mode', 'c1_mode', c2_mode', 'c3_mode' and 'all_mode'. Defaults to 'all_mode'. 
        component_mode (str, optional): Component mode of the FFmpeg tblend. Available values for component modes can be accessed here: https://ffmpeg.org/ffmpeg-filters.html#blend-1. Defaults to 'average'.
        target_name (str, optional): The name of the output video. Defaults to None (which assumes that the input filename with the component mode suffix should be used).
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgImage: A new MgImage pointing to the output image file.
    """

    if filename is None:
        filename = self.filename

    of, fex = os.path.splitext(filename)

    if target_name is None:
        target_name = of + f'_{component_mode}.png'
    if not overwrite:
        target_name = generate_outfilename(target_name)

    # Get the number of frames
    frames = get_framecount(filename)
    # Get the number of times all frames can be divided
    divider = int(np.ceil(np.log(frames / 2) / np.log(2)))

    # Define ffmpeg command
    cmd = ['ffmpeg', '-y', '-i', filename]

    cmd_filter = ''
    # Set average blur
    if self.blur.lower() == 'average':
        cmd_filter += 'avgblur=sizeX=10:sizeY=10,'
    # set color mode
    if self.color == True:
        pixformat = 'gbrp'
    else:
        pixformat = 'gray'
    cmd_filter += f'format={pixformat},'

    # Set frame blend every two frames
    cmd_filter += f'tblend={mode}={component_mode},framestep=2,' * divider + 'setpts=1*PTS'
    cmd_end = ['-frames:v', '1', target_name]
    cmd += ['-vf', cmd_filter] + cmd_end

    # Run the command using ffmpeg and wait for it to finish
    ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering blended image:')

    # Save result as the blended image for parent MgObject
    self.blend_image = MgImage(target_name)

    return self.blend_image

pixelarray

pixelarray(width=640, target_name=None, overwrite=True)

Creates a 'Frame-Averaged Pixel Array' of a video by reducing each frame to a single pixel and arranging all frames into a single image. This is equivalent to the bash script that scales each frame to 1x1 pixel and then tiles them into a grid.

Based on the original bash script concept: - Each frame is reduced to a single pixel (average color of the frame) - All pixel values are arranged in a grid with specified width - Height is calculated automatically based on total frames and width

Parameters:

Name	Type	Description	Default
width	int	Width of the output image in pixels (number of frame-pixels per row). Defaults to 640.	640
target_name	str	The name of the output image file. If None, uses input filename with 'framearray' suffix. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgImage		A new MgImage pointing to the output frame-averaged pixel array image file.

Source code in musicalgestures/_frameaverage.py

def mg_pixelarray(self, width=640, target_name=None, overwrite=True):
    """
    Creates a 'Frame-Averaged Pixel Array' of a video by reducing each frame to a single pixel
    and arranging all frames into a single image. This is equivalent to the bash script that
    scales each frame to 1x1 pixel and then tiles them into a grid.

    Based on the original bash script concept:
    - Each frame is reduced to a single pixel (average color of the frame)
    - All pixel values are arranged in a grid with specified width
    - Height is calculated automatically based on total frames and width

    Args:
        width (int, optional): Width of the output image in pixels (number of frame-pixels per row). 
                              Defaults to 640.
        target_name (str, optional): The name of the output image file. If None, uses input filename 
                                   with '_framearray_<width>' suffix. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically 
                                  increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgImage: A new MgImage pointing to the output frame-averaged pixel array image file.
    """

    target_name = resolve_filename(self.of, f"_pixelarray_{width}.png", target_name, overwrite)

    # Get video properties
    frames = get_framecount(self.filename)
    height = int(np.ceil(frames / width))
    video_length = get_length(self.filename)

    # Method 1: Using FFmpeg (similar to the bash script)
    # This directly replicates the bash script functionality
    cmd = [
        'ffmpeg', '-y', '-i', self.filename,
        '-vf', f'scale=1:1,tile={width}x{height}',
        '-frames:v', '1',
        target_name
    ]

    ffmpeg_cmd(cmd, video_length, pb_prefix='Creating frame-averaged pixel array:')

    # Save result as the pixelarray for parent MgVideo
    self.pixelarray = MgImage(target_name)

    return self.pixelarray

pixelarray_cv2

pixelarray_cv2(width=640, target_name=None, overwrite=True)

Alternative implementation using OpenCV for more control over the process. Creates a 'Frame-Averaged Pixel Array' by reading each frame, calculating its average color, and arranging these average colors in a grid.

Parameters:

Name	Type	Description	Default
width	int	Width of the output image in pixels. Defaults to 640.	640
target_name	str	The name of the output image file. Defaults to None.	None
overwrite	bool	Whether to allow overwriting existing files. Defaults to True.	True

Returns:

Name	Type	Description
MgImage		A new MgImage pointing to the output frame-averaged pixel array image file.

Source code in musicalgestures/_frameaverage.py

def mg_pixelarray_cv2(self, width=640, target_name=None, overwrite=True):
    """
    Alternative implementation using OpenCV for more control over the process.
    Creates a 'Frame-Averaged Pixel Array' by reading each frame, calculating its average color,
    and arranging these average colors in a grid.

    Args:
        width (int, optional): Width of the output image in pixels. Defaults to 640.
        target_name (str, optional): The name of the output image file. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files. Defaults to True.

    Returns:
        MgImage: A new MgImage pointing to the output frame-averaged pixel array image file.
    """

    target_name = resolve_filename(self.of, f"_pixelarray_cv2_{width}.png", target_name, overwrite)

    # Open video
    cap = cv2.VideoCapture(self.filename)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

    # Calculate output dimensions
    height = int(np.ceil(total_frames / width))

    pb = MgProgressbar(total=total_frames, prefix='Creating frame-averaged pixel array (cv2):')

    # Create output array
    if self.color:
        output_array = np.zeros((height, width, 3), dtype=np.uint8)
    else:
        output_array = np.zeros((height, width), dtype=np.uint8)

    frame_count = 0

    try:
        while True:
            ret, frame = cap.read()
            if not ret:
                break

            # Convert to grayscale if needed
            if not self.color:
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
                average_color = np.mean(frame)
            else:
                # Calculate average color for each channel
                average_color = np.mean(frame, axis=(0, 1))

            # Calculate position in output grid
            row = frame_count // width
            col = frame_count % width

            # Only process if within our output bounds
            if row < height:
                output_array[row, col] = average_color.astype(np.uint8)

            frame_count += 1
            pb.progress(frame_count)

    finally:
        cap.release()

    # Save the image
    cv2.imwrite(target_name, output_array)

    # Save result as the pixelarray_cv2 for parent MgVideo
    self.pixelarray_cv2 = MgImage(target_name)

    return self.pixelarray_cv2

pixelarray_stats

pixelarray_stats(width=640, include_stats=True)

Creates a frame-averaged pixel array and optionally returns statistics about the video. This function provides additional information similar to the bash script's output.

Parameters:

Name	Type	Description	Default
width	int	Width of the output image in pixels. Defaults to 640.	640
include_stats	bool	Whether to return detailed statistics. Defaults to True.	True

Returns:

Name	Type	Description
dict		Dictionary containing the generated MgImage and optional statistics.

Source code in musicalgestures/_frameaverage.py

def mg_pixelarray_stats(self, width=640, include_stats=True):
    """
    Creates a frame-averaged pixel array and optionally returns statistics about the video.
    This function provides additional information similar to the bash script's output.

    Args:
        width (int, optional): Width of the output image in pixels. Defaults to 640.
        include_stats (bool, optional): Whether to return detailed statistics. Defaults to True.

    Returns:
        dict: Dictionary containing the generated MgImage and optional statistics.
    """

    # Get video properties for statistics (similar to bash script)
    cap = cv2.VideoCapture(self.filename)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    fps = cap.get(cv2.CAP_PROP_FPS)
    duration_seconds = total_frames / fps if fps > 0 else 0
    cap.release()

    # Calculate dimensions
    height = int(np.ceil(total_frames / width))

    # Create the frame-averaged pixel array
    result_image = mg_pixelarray(self, width=width)

    result = {
        'image': result_image,
        'filename': os.path.abspath(self.filename)
    }

    if include_stats:
        # Format duration as HH:MM:SS.ms
        hours = int(duration_seconds // 3600)
        minutes = int((duration_seconds % 3600) // 60)
        seconds = duration_seconds % 60
        duration_str = f"{hours:02d}:{minutes:02d}:{seconds:06.3f}"

        result.update({
            'duration': duration_str,
            'duration_seconds': int(duration_seconds),
            'fps': int(fps + 0.5),
            'total_frames': total_frames,
            'output_width': width,
            'output_height': height,
            'filter_description': f"scale=1:1,tile={width}x{height}"
        })

    return result

heatmap

heatmap(colormap='inferno', overlay=True, alpha=0.75, background_dim=0.4, blur=0, normalize=True, gamma=0.5, target_name=None, overwrite=True)

Renders a motion heatmap showing which parts of the video change the most.

The function accumulates the absolute pixel difference between consecutive frames over the whole video, producing a single image where bright/hot regions mark areas of frequent or large change and dark/cool regions mark areas that stay still. When overlay is True the heat is composited on top of a dimmed average frame, so the activity is shown in the spatial context of the scene.

Parameters:

Name	Type	Description	Default
colormap	str	Any matplotlib colormap name used to colour the heat (e.g. 'inferno', 'jet', 'viridis', 'hot', 'magma'). Defaults to 'inferno'.	'inferno'
overlay	bool	If True, composite the heatmap over a dimmed grayscale average frame so the motion is shown in context. If False, render the bare heatmap on a black background. Defaults to True.	True
alpha	float	Maximum opacity of the heat overlay in [0, 1]. Hotter pixels are more opaque. Only used when overlay=True. Defaults to 0.75.	0.75
background_dim	float	Brightness multiplier for the average-frame background in [0, 1]. Lower values make the heat stand out more. Only used when overlay=True. Defaults to 0.4.	0.4
blur	int	Radius of an optional Gaussian smoothing applied to the accumulated motion (0 disables). Gives a smoother, less speckled heatmap. Defaults to 0.	0
normalize	bool	If True, scale the accumulated motion so the most active pixel maps to the top of the colormap. Defaults to True.	True
gamma	float	Gamma applied to the normalised heat before colouring. Values < 1 boost faint motion so subtle activity is visible; 1.0 is linear. Defaults to 0.5.	0.5
target_name	str	The name of the output image. Defaults to None (which uses the input filename with the suffix "_heatmap.png").	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment the target filename to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	A new MgImage pointing to the output heatmap image file.

Source code in musicalgestures/_heatmap.py

def mg_heatmap(
        self,
        colormap: str = 'inferno',
        overlay: bool = True,
        alpha: float = 0.75,
        background_dim: float = 0.4,
        blur: int = 0,
        normalize: bool = True,
        gamma: float = 0.5,
        target_name: str | None = None,
        overwrite: bool = True) -> "MgImage":
    """
    Renders a motion heatmap showing which parts of the video change the most.

    The function accumulates the absolute pixel difference between consecutive frames
    over the whole video, producing a single image where bright/hot regions mark areas
    of frequent or large change and dark/cool regions mark areas that stay still. When
    ``overlay`` is True the heat is composited on top of a dimmed average frame, so the
    activity is shown in the spatial context of the scene.

    Args:
        colormap (str, optional): Any matplotlib colormap name used to colour the heat
            (e.g. 'inferno', 'jet', 'viridis', 'hot', 'magma'). Defaults to 'inferno'.
        overlay (bool, optional): If True, composite the heatmap over a dimmed grayscale
            average frame so the motion is shown in context. If False, render the bare
            heatmap on a black background. Defaults to True.
        alpha (float, optional): Maximum opacity of the heat overlay in [0, 1]. Hotter
            pixels are more opaque. Only used when ``overlay=True``. Defaults to 0.75.
        background_dim (float, optional): Brightness multiplier for the average-frame
            background in [0, 1]. Lower values make the heat stand out more. Only used
            when ``overlay=True``. Defaults to 0.4.
        blur (int, optional): Radius of an optional Gaussian smoothing applied to the
            accumulated motion (0 disables). Gives a smoother, less speckled heatmap.
            Defaults to 0.
        normalize (bool, optional): If True, scale the accumulated motion so the most
            active pixel maps to the top of the colormap. Defaults to True.
        gamma (float, optional): Gamma applied to the normalised heat before colouring.
            Values < 1 boost faint motion so subtle activity is visible; 1.0 is linear.
            Defaults to 0.5.
        target_name (str, optional): The name of the output image. Defaults to None
            (which uses the input filename with the suffix "_heatmap.png").
        overwrite (bool, optional): Whether to allow overwriting existing files or to
            automatically increment the target filename to avoid overwriting.
            Defaults to True.

    Returns:
        MgImage: A new MgImage pointing to the output heatmap image file.
    """

    target_name = resolve_filename(self.of, '_heatmap.png', target_name, overwrite)

    cap = cv2.VideoCapture(self.filename)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    accum = np.zeros((height, width), dtype=np.float64)      # accumulated motion
    bg_accum = np.zeros((height, width, 3), dtype=np.float64)  # for the average frame (RGB)
    prev_gray = None
    n = 0

    pb = MgProgressbar(total=total_frames, prefix='Rendering motion heatmap:')

    try:
        while True:
            ret, frame = cap.read()
            if not ret:
                break
            bg_accum += frame[..., ::-1].astype(np.float64)  # BGR -> RGB
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
            if prev_gray is not None:
                accum += np.abs(gray - prev_gray)
            prev_gray = gray
            n += 1
            pb.progress(n)
    finally:
        cap.release()

    pb.progress(total_frames)

    if n == 0:
        raise RuntimeError(f"Could not read any frames from {self.filename}.")

    avg_frame = (bg_accum / n).astype(np.uint8)

    # Optional smoothing of the accumulated motion
    if blur and blur > 0:
        k = int(blur) * 2 + 1
        accum = cv2.GaussianBlur(accum, (k, k), 0)

    # Normalise to [0, 1]
    if normalize and accum.max() > 0:
        heat = accum / accum.max()
    else:
        heat = np.clip(accum, 0, 1)

    # Gamma boost so subtle motion stays visible
    if gamma and gamma != 1.0:
        heat = np.power(heat, gamma)

    # Colour-map the heat (RGB, uint8)
    cmap = matplotlib.colormaps[colormap]
    heat_rgb = (cmap(heat)[..., :3] * 255).astype(np.float64)

    if overlay:
        # Dimmed grayscale average frame as background
        gray_bg = cv2.cvtColor(avg_frame, cv2.COLOR_RGB2GRAY)
        gray_bg3 = np.stack([gray_bg] * 3, axis=-1).astype(np.float64) * background_dim
        # Hotter pixels are more opaque
        a = (heat[..., None] * alpha)
        out = gray_bg3 * (1 - a) + heat_rgb * a
    else:
        out = heat_rgb

    out = np.clip(out, 0, 255).astype(np.uint8)

    # cv2 writes BGR
    cv2.imwrite(target_name, out[..., ::-1])

    # NB: store under a non-shadowing attribute so the heatmap() method stays callable
    self.heatmap_image = MgImage(target_name)
    return self.heatmap_image

motiontempo

motiontempo(fmin=0.2, fmax=8.0, dpi=300, autoshow=True, title=None, target_name=None, overwrite=True)

Estimates the dominant movement tempo of a video from its quantity of motion.

A quantity-of-motion (QoM) signal is computed as the mean absolute difference between consecutive frames. Its dominant periodicity within [fmin, fmax] is found with an FFT and reported both in Hz and in beats per minute (BPM), giving a simple estimate of the overall movement tempo (e.g. step rate of a dancer).

Parameters:

Name	Type	Description	Default
fmin	float	Lowest movement frequency to consider (Hz). Defaults to 0.2.	0.2
fmax	float	Highest movement frequency to consider (Hz). Defaults to 8.0.	8.0
dpi	int	Image quality of the rendered figure in DPI. Defaults to 300.	300
autoshow	bool	Whether to show the resulting figure automatically. Defaults to True.	True
title	str	Optionally add a title to the figure. Use 'filename' for the file name. Defaults to None.	None
target_name	str	The name of the output image. Defaults to None (which uses the input filename with the suffix "_motiontempo.png").	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment the target filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	MgFigure	An MgFigure object. Numeric results are available in .data: 'tempo_bpm', 'dominant_frequency', 'qom', 'times', 'freqs', 'spectrum', 'fps'.

Source code in musicalgestures/_motiontempo.py

def mg_motiontempo(self, fmin=0.2, fmax=8.0, dpi=300, autoshow=True, title=None, target_name=None, overwrite=True) -> "MgFigure":
    """
    Estimates the dominant movement tempo of a video from its quantity of motion.

    A quantity-of-motion (QoM) signal is computed as the mean absolute difference
    between consecutive frames. Its dominant periodicity within ``[fmin, fmax]`` is
    found with an FFT and reported both in Hz and in beats per minute (BPM), giving a
    simple estimate of the overall movement tempo (e.g. step rate of a dancer).

    Args:
        fmin (float, optional): Lowest movement frequency to consider (Hz). Defaults to 0.2.
        fmax (float, optional): Highest movement frequency to consider (Hz). Defaults to 8.0.
        dpi (int, optional): Image quality of the rendered figure in DPI. Defaults to 300.
        autoshow (bool, optional): Whether to show the resulting figure automatically. Defaults to True.
        title (str, optional): Optionally add a title to the figure. Use 'filename' for the file name. Defaults to None.
        target_name (str, optional): The name of the output image. Defaults to None
            (which uses the input filename with the suffix "_motiontempo.png").
        overwrite (bool, optional): Whether to allow overwriting existing files or to
            automatically increment the target filename. Defaults to True.

    Returns:
        MgFigure: An MgFigure object. Numeric results are available in ``.data``:
            'tempo_bpm', 'dominant_frequency', 'qom', 'times', 'freqs', 'spectrum', 'fps'.
    """
    from musicalgestures._analysis import dominant_frequency

    target_name = resolve_filename(self.of, '_motiontempo.png', target_name, overwrite)

    cap = cv2.VideoCapture(self.filename)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    fps = cap.get(cv2.CAP_PROP_FPS) or self.fps

    qom = []
    prev_gray = None
    pb = MgProgressbar(total=total_frames, prefix='Computing movement tempo:')
    n = 0
    try:
        while True:
            ret, frame = cap.read()
            if not ret:
                break
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
            if prev_gray is not None:
                qom.append(float(np.abs(gray - prev_gray).mean()))
            prev_gray = gray
            n += 1
            pb.progress(n)
    finally:
        cap.release()
    pb.progress(total_frames)

    qom = np.asarray(qom, dtype=float)
    if len(qom) < 4:
        raise RuntimeError(f"Not enough frames in {self.filename} to estimate movement tempo.")

    # Normalise QoM to [0, 1] by pixel value range (mean abs frame difference / 255)
    qom = qom / 255.0
    mean_qom = float(qom.mean())

    times = np.arange(len(qom)) / fps

    # Dominant movement frequency and its spectrum within [fmin, fmax]
    dom_freq = dominant_frequency(qom, fps, fmin=fmin, fmax=fmax)
    tempo_bpm = dom_freq * 60.0

    freqs = np.fft.rfftfreq(len(qom), d=1.0 / fps)
    spectrum = np.abs(np.fft.rfft(qom - qom.mean()))

    fig, ax = plt.subplots(nrows=2, figsize=(12, 6), dpi=dpi)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    if title is None:
        title = ''
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title, fontsize=16)

    ax[0].plot(times, qom, color='#1f77b4', lw=0.8)
    # Average quantity of motion line + number
    ax[0].axhline(mean_qom, color='#d62728', ls='--', lw=1.2,
                  label=f'avg QoM = {mean_qom:.3f}')
    ax[0].legend(loc='upper right')
    ax[0].set(title='Quantity of motion', xlabel='Time (s)', ylabel='QoM (normalised 0–1)')
    ax[0].set_xlim(0, times[-1] if len(times) else 1)
    ax[0].set_ylim(0, max(qom.max() * 1.05, 1e-6))

    mask = (freqs >= fmin) & (freqs <= fmax)
    ax[1].plot(freqs[mask], spectrum[mask], color='#ff7f0e', lw=0.9)
    # Average beat frequency line + number
    if dom_freq > 0:
        ax[1].axvline(dom_freq, color='r', ls='--', lw=1.2,
                      label=f'avg beat frequency = {dom_freq:.2f} Hz ({tempo_bpm:.1f} BPM)')
        ax[1].legend(loc='upper right')
    ax[1].set(title='Motion spectrum', xlabel='Frequency (Hz)', ylabel='Magnitude')
    ax[1].set_xlim(fmin, fmax)

    plt.tight_layout(rect=[0, 0, 1, 0.96])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    data = {
        'tempo_bpm': tempo_bpm,
        'dominant_frequency': dom_freq,
        'mean_qom': mean_qom,
        'qom': qom,
        'times': times,
        'freqs': freqs,
        'spectrum': spectrum,
        'fps': fps,
    }

    mgf = MgFigure(
        figure=fig,
        figure_type='video.motiontempo',
        data=data,
        layers=None,
        image=target_name)

    return mgf

motionvectors

motionvectors(target_name=None, overwrite=True)

Renders a video visualising the motion vectors encoded in the input video.

Inter-frame codecs (MPEG-1/2/4, H.264, H.265, …) store motion vectors that describe how macroblocks move between frames. This method uses FFmpeg's codecview filter (with -flags2 +export_mvs) to draw those vectors as arrows on top of the video, giving a quick, decoder-level view of motion without any re-computation.

NB: Only codecs that actually carry motion vectors will show arrows. Intra-only formats (e.g. MJPEG, common in .avi files) have none — convert to an inter-frame codec first (e.g. via show(mode='notebook') which makes an mp4, or any mp4/h264 source) to see motion vectors.

Parameters:

Name	Type	Description	Default
target_name	str	Target output name for the video. Defaults to None (which uses the input filename with the suffix "_motionvectors").	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment the target filename. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	MgVideo	An MgVideo pointing to the rendered motion-vector video.

Source code in musicalgestures/_motionvectors.py

def mg_motionvectors(self, target_name=None, overwrite=True) -> "musicalgestures.MgVideo":
    """
    Renders a video visualising the motion vectors encoded in the input video.

    Inter-frame codecs (MPEG-1/2/4, H.264, H.265, …) store motion vectors that describe
    how macroblocks move between frames. This method uses FFmpeg's ``codecview`` filter
    (with ``-flags2 +export_mvs``) to draw those vectors as arrows on top of the video,
    giving a quick, decoder-level view of motion without any re-computation.

    NB: Only codecs that actually carry motion vectors will show arrows. Intra-only
    formats (e.g. MJPEG, common in ``.avi`` files) have none — convert to an inter-frame
    codec first (e.g. via ``show(mode='notebook')`` which makes an mp4, or any mp4/h264
    source) to see motion vectors.

    Args:
        target_name (str, optional): Target output name for the video. Defaults to None
            (which uses the input filename with the suffix "_motionvectors").
        overwrite (bool, optional): Whether to allow overwriting existing files or to
            automatically increment the target filename. Defaults to True.

    Returns:
        MgVideo: An MgVideo pointing to the rendered motion-vector video.
    """
    of, fex = os.path.splitext(self.filename)

    target_name = resolve_filename(of, '_motionvectors' + fex, target_name, overwrite)

    # -flags2 +export_mvs must precede -i so the decoder exports motion vectors;
    # codecview then draws them (pf=P-frame forward, bf/bb=B-frame forward/backward).
    cmd = [
        'ffmpeg', '-y', '-flags2', '+export_mvs', '-i', self.filename,
        '-vf', 'codecview=mv=pf+bf+bb', '-q:v', '3', target_name,
    ]
    ffmpeg_cmd(cmd, get_length(self.filename), pb_prefix='Rendering motion vectors:')

    self.motionvectors_video = musicalgestures.MgVideo(
        target_name, color=self.color, returned_by_process=True)
    return self.motionvectors_video

eulerian

eulerian(mode='color', freq_low=0.83, freq_high=1.0, amplification=50, levels=4, chroma_attenuation=1.0, lambda_cutoff=16, target_name=None, overwrite=True)

Applies Eulerian Video Magnification (EVM) to reveal subtle changes in a video.

EVM amplifies small temporal variations that are normally invisible. Two modes are available:

• mode='color' — amplifies subtle colour changes (e.g. blood flow / pulse, breathing). Uses a Gaussian pyramid and an ideal (FFT) temporal band-pass filter. Processed in two passes so only a small down-sampled stack is held in memory.
• mode='motion' — amplifies subtle motion. Uses a Laplacian pyramid with a streaming IIR temporal band-pass filter and spatial-wavelength attenuation, so it runs frame-by-frame with low memory use.

Based on Wu et al., "Eulerian Video Magnification for Revealing Subtle Changes in the World" (SIGGRAPH 2012).

Parameters:

Name	Type	Description	Default
mode	str	'color' or 'motion'. Defaults to 'color'.	'color'
freq_low	float	Lower temporal cutoff in Hz. Defaults to 0.83 (~50 bpm).	0.83
freq_high	float	Upper temporal cutoff in Hz. Defaults to 1.0 (~60 bpm).	1.0
amplification	float	Amplification factor (alpha). Defaults to 50.	50
levels	int	Number of spatial pyramid levels. Defaults to 4.	4
chroma_attenuation	float	Chrominance attenuation in [0, 1] (color mode). Lower values reduce colour artefacts. Defaults to 1.0.	1.0
lambda_cutoff	float	Spatial wavelength cutoff for amplitude attenuation (motion mode). Defaults to 16.	16
target_name	str	Target output name. Defaults to None (input filename with the suffix "_evm").	None
overwrite	bool	Whether to allow overwriting or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	MgVideo	An MgVideo pointing to the magnified output video.

Source code in musicalgestures/_eulerian.py

def mg_eulerian(
        self,
        mode='color',
        freq_low=0.83,
        freq_high=1.0,
        amplification=50,
        levels=4,
        chroma_attenuation=1.0,
        lambda_cutoff=16,
        target_name=None,
        overwrite=True) -> "musicalgestures.MgVideo":
    """
    Applies Eulerian Video Magnification (EVM) to reveal subtle changes in a video.

    EVM amplifies small temporal variations that are normally invisible. Two modes are
    available:

    * ``mode='color'`` — amplifies subtle **colour** changes (e.g. blood flow / pulse,
      breathing). Uses a Gaussian pyramid and an ideal (FFT) temporal band-pass filter.
      Processed in two passes so only a small down-sampled stack is held in memory.
    * ``mode='motion'`` — amplifies subtle **motion**. Uses a Laplacian pyramid with a
      streaming IIR temporal band-pass filter and spatial-wavelength attenuation, so it
      runs frame-by-frame with low memory use.

    Based on Wu et al., "Eulerian Video Magnification for Revealing Subtle Changes in the
    World" (SIGGRAPH 2012).

    Args:
        mode (str, optional): 'color' or 'motion'. Defaults to 'color'.
        freq_low (float, optional): Lower temporal cutoff in Hz. Defaults to 0.83 (~50 bpm).
        freq_high (float, optional): Upper temporal cutoff in Hz. Defaults to 1.0 (~60 bpm).
        amplification (float, optional): Amplification factor (alpha). Defaults to 50.
        levels (int, optional): Number of spatial pyramid levels. Defaults to 4.
        chroma_attenuation (float, optional): Chrominance attenuation in [0, 1] (color mode).
            Lower values reduce colour artefacts. Defaults to 1.0.
        lambda_cutoff (float, optional): Spatial wavelength cutoff for amplitude attenuation
            (motion mode). Defaults to 16.
        target_name (str, optional): Target output name. Defaults to None (input filename with
            the suffix "_evm").
        overwrite (bool, optional): Whether to allow overwriting or auto-increment the filename.
            Defaults to True.

    Returns:
        MgVideo: An MgVideo pointing to the magnified output video.
    """
    of, fex = os.path.splitext(self.filename)
    target_name = resolve_filename(of, '_evm' + fex, target_name, overwrite)

    mode = mode.lower()
    width, height, fps = self.width, self.height, self.fps
    # NB: for MgVideo, self.length is the frame count, not seconds.
    n_frames = self.length
    duration_s = self.length / fps if fps else 0

    def _open_reader():
        cmd = ['ffmpeg', '-y', '-i', self.filename]
        return ffmpeg_cmd(cmd, total_time=duration_s, pipe='read')

    def _open_writer():
        cmd = ['ffmpeg', '-y', '-s', f'{width}x{height}', '-r', str(fps), '-f', 'rawvideo',
               '-pix_fmt', 'bgr24', '-vcodec', 'rawvideo', '-i', '-',
               '-vcodec', 'libx264', '-pix_fmt', 'yuv420p', target_name]
        return ffmpeg_cmd(cmd, total_time=duration_s, pipe='write')

    frame_bytes = width * height * 3

    if mode == 'color':
        # ---- Pass 1: collect the small Gaussian level for every frame ----
        pb = MgProgressbar(total=n_frames * 2, prefix='EVM (color):')
        process = _open_reader()
        small_stack = []
        i = 0
        while True:
            buf = process.stdout.read(frame_bytes)
            if len(buf) < frame_bytes:
                break
            frame = np.frombuffer(buf, dtype=np.uint8).reshape(height, width, 3).astype(np.float32)
            small_stack.append(_gaussian_top(frame, levels))
            i += 1
            pb.progress(i)

        if len(small_stack) < 2:
            raise RuntimeError(f"Not enough frames in {self.filename} for EVM.")

        stack = np.stack(small_stack, axis=0)
        del small_stack

        # Temporal band-pass + amplification
        filtered = _ideal_bandpass(stack, fps, freq_low, freq_high)
        filtered[..., 0] *= amplification                       # B
        filtered[..., 1] *= amplification * chroma_attenuation  # G
        filtered[..., 2] *= amplification * chroma_attenuation  # R
        del stack

        # ---- Pass 2: add upsampled magnified signal back, write out ----
        process = _open_reader()
        writer = _open_writer()
        i = 0
        while True:
            buf = process.stdout.read(frame_bytes)
            if len(buf) < frame_bytes:
                break
            frame = np.frombuffer(buf, dtype=np.uint8).reshape(height, width, 3).astype(np.float32)
            mag = cv2.resize(filtered[i], (width, height))
            out = np.clip(frame + mag, 0, 255).astype(np.uint8)
            writer.stdin.write(out.tobytes())
            i += 1
            pb.progress(n_frames + i)
        writer.stdin.close()
        writer.wait()
        pb.progress(n_frames * 2)

    elif mode == 'motion':
        # ---- Streaming Laplacian-pyramid IIR band-pass ----
        pb = MgProgressbar(total=n_frames, prefix='EVM (motion):')
        process = _open_reader()
        writer = _open_writer()

        lowpass1 = None
        lowpass2 = None
        # IIR cutoffs derived from the requested band
        r1 = freq_high * 2.0 / fps
        r2 = freq_low * 2.0 / fps
        exaggeration = 2.0
        i = 0
        while True:
            buf = process.stdout.read(frame_bytes)
            if len(buf) < frame_bytes:
                break
            frame = np.frombuffer(buf, dtype=np.uint8).reshape(height, width, 3).astype(np.float32)
            pyr = _laplacian_pyramid(frame, levels)

            if lowpass1 is None:
                lowpass1 = [lvl.copy() for lvl in pyr]
                lowpass2 = [lvl.copy() for lvl in pyr]
                filtered = [np.zeros_like(lvl) for lvl in pyr]
            else:
                for k in range(len(pyr)):
                    lowpass1[k] = (1 - r1) * lowpass1[k] + r1 * pyr[k]
                    lowpass2[k] = (1 - r2) * lowpass2[k] + r2 * pyr[k]
                    filtered[k] = lowpass1[k] - lowpass2[k]

            # Spatial-wavelength dependent amplification (attenuate top & bottom levels)
            delta = lambda_cutoff / 8.0 / (1.0 + amplification)
            lambda_repr = (width ** 2 + height ** 2) ** 0.5 / 3.0
            amplified = []
            for k in range(len(pyr)):
                if k == 0 or k == len(pyr) - 1:
                    amplified.append(np.zeros_like(pyr[k]))
                else:
                    curr_alpha = lambda_repr / delta / 8.0 - 1.0
                    curr_alpha *= exaggeration
                    alpha_k = min(amplification, max(0.0, curr_alpha))
                    amplified.append(pyr[k] + filtered[k] * alpha_k)
                lambda_repr /= 2.0
            # Keep the residual (bottom) and finest (top) unamplified for stability
            amplified[-1] = pyr[-1]
            amplified[0] = pyr[0]

            out = _reconstruct_from_laplacian(amplified)
            out = np.clip(out, 0, 255).astype(np.uint8)
            writer.stdin.write(out.tobytes())
            i += 1
            pb.progress(i)

        writer.stdin.close()
        writer.wait()
        pb.progress(n_frames)

    else:
        raise ValueError(f"Unknown mode '{mode}'. Use 'color' or 'motion'.")

    self.eulerian_video = musicalgestures.MgVideo(target_name, color=self.color, returned_by_process=True)
    return self.eulerian_video

sonomotiongram

sonomotiongram(sonogram='vertical', n_fft=2048, sr=22050, n_iter=32, flip=True, normalize=True, target_name=None, overwrite=True)

Creates a sonomotiongram: a sonification of the video's motiongram.

The motiongram (a time–space image of where motion happens) is treated as a magnitude spectrogram — spatial position maps to frequency, motion intensity to amplitude — and converted back to audio with an inverse STFT (Griffin–Lim phase estimation). The result lets you hear the motion. Based on Jensenius, "Some video abstraction techniques for displaying body movement in analysis and performance" / sonomotiongrams (SMC 2013).

Parameters:

Name	Type	Description	Default
sonogram	str	Which motiongram to sonify: 'vertical' (motion across the vertical axis) or 'horizontal'. Defaults to 'vertical'.	'vertical'
n_fft	int	FFT size; sets the number of frequency bins (n_fft//2+1) the motiongram rows are mapped onto. Defaults to 2048.	2048
sr	int	Sample rate of the rendered audio. Defaults to 22050.	22050
n_iter	int	Griffin–Lim iterations for phase estimation (higher = cleaner, slower). Defaults to 32.	32
flip	bool	If True, map the top of the image to high frequencies (usually more intuitive). Defaults to True.	True
normalize	bool	Normalise the rendered audio to peak 1.0. Defaults to True.	True
target_name	str	Output audio filename. Defaults to None (input filename with the suffix "sono.wav").	None
overwrite	bool	Whether to allow overwriting or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgAudio		An MgAudio pointing to the rendered sonification (WAV).

Source code in musicalgestures/_sonification.py

def mg_sonomotiongram(
        self,
        sonogram='vertical',
        n_fft=2048,
        sr=22050,
        n_iter=32,
        flip=True,
        normalize=True,
        target_name=None,
        overwrite=True):
    """
    Creates a *sonomotiongram*: a sonification of the video's motiongram.

    The motiongram (a time–space image of where motion happens) is treated as a magnitude
    spectrogram — spatial position maps to frequency, motion intensity to amplitude — and
    converted back to audio with an inverse STFT (Griffin–Lim phase estimation). The result
    lets you *hear* the motion. Based on Jensenius, "Some video abstraction techniques for
    displaying body movement in analysis and performance" / sonomotiongrams (SMC 2013).

    Args:
        sonogram (str, optional): Which motiongram to sonify: 'vertical' (motion across the
            vertical axis) or 'horizontal'. Defaults to 'vertical'.
        n_fft (int, optional): FFT size; sets the number of frequency bins (n_fft//2+1) the
            motiongram rows are mapped onto. Defaults to 2048.
        sr (int, optional): Sample rate of the rendered audio. Defaults to 22050.
        n_iter (int, optional): Griffin–Lim iterations for phase estimation (higher = cleaner,
            slower). Defaults to 32.
        flip (bool, optional): If True, map the top of the image to high frequencies (usually
            more intuitive). Defaults to True.
        normalize (bool, optional): Normalise the rendered audio to peak 1.0. Defaults to True.
        target_name (str, optional): Output audio filename. Defaults to None (input filename
            with the suffix "_sono_<sonogram>.wav").
        overwrite (bool, optional): Whether to allow overwriting or auto-increment the filename.
            Defaults to True.

    Returns:
        MgAudio: An MgAudio pointing to the rendered sonification (WAV).
    """
    import librosa

    sonogram = sonogram.lower()
    if sonogram not in ('vertical', 'horizontal'):
        raise ValueError("sonogram must be 'vertical' or 'horizontal'.")

    target_name = resolve_filename(self.of, f"_sono_{sonogram}.wav", target_name, overwrite)

    width, height, fps = self.width, self.height, self.fps
    # NB: for MgVideo, self.length is the frame count, not seconds.
    duration_s = self.length / fps if fps else 0
    frame_bytes = width * height * 3

    # --- Build the motiongram (magnitude over time) from frame differences ---
    cmd = ['ffmpeg', '-y', '-i', self.filename]
    process = ffmpeg_cmd(cmd, total_time=duration_s, pipe='read')
    pb = MgProgressbar(total=self.length, prefix='Building motiongram for sonification:')

    columns = []
    prev_gray = None
    n = 0
    while True:
        buf = process.stdout.read(frame_bytes)
        if len(buf) < frame_bytes:
            break
        frame = np.frombuffer(buf, dtype=np.uint8).reshape(height, width, 3)
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
        if prev_gray is not None:
            motion = np.abs(gray - prev_gray)
            if sonogram == 'vertical':
                columns.append(motion.mean(axis=1))   # length H (vertical position)
            else:
                columns.append(motion.mean(axis=0))   # length W (horizontal position)
        prev_gray = gray
        n += 1
        pb.progress(n)
    pb.progress(self.length)

    if len(columns) < 2:
        raise RuntimeError(f"Not enough frames in {self.filename} to build a sonomotiongram.")

    gram = np.stack(columns, axis=1)  # shape (space, time)
    if flip:
        gram = gram[::-1, :]

    # --- Treat the motiongram as a magnitude spectrogram and invert it ---
    n_freq = n_fft // 2 + 1
    n_time = gram.shape[1]
    # Map spatial rows -> frequency bins
    mag = cv2.resize(gram.astype(np.float32), (n_time, n_freq), interpolation=cv2.INTER_LINEAR)

    # Scale magnitudes to a useful range
    if mag.max() > 0:
        mag = mag / mag.max()

    # Choose hop so the audio duration matches the video duration
    total_samples = int(sr * duration_s)
    hop_length = max(1, total_samples // n_time)

    y = librosa.griffinlim(mag, n_iter=n_iter, hop_length=hop_length, win_length=n_fft, n_fft=n_fft)

    if normalize:
        peak = np.max(np.abs(y))
        if peak > 0:
            y = y / peak

    # Write the WAV
    try:
        import soundfile as sf
        sf.write(target_name, y.astype(np.float32), sr)
    except Exception:
        from scipy.io import wavfile
        wavfile.write(target_name, sr, (y * 32767).astype(np.int16))

    # Save the result as sonomotiongram_audio for the parent MgVideo. NB: not
    # `self.sonomotiongram`, which would overwrite (shadow) the bound method
    # and break any subsequent sonomotiongram() call on the same object.
    self.sonomotiongram_audio = musicalgestures.MgAudio(target_name, sr=sr)
    return self.sonomotiongram_audio

stroboscope

stroboscope(n_samples=12, method='auto', threshold=0.1, kernel_size=5, keep_largest=False, colorize=True, background='average', target_name=None, overwrite=True)

Renders a stroboscope / chronophotography image: the person's silhouette at evenly sampled times composited onto a single frame, showing the body moving through space over time (Muybridge-style).

For a clean result with a single person on a static background, raise threshold and set keep_largest=True so only the person's blob is composited (avoids the image "blowing up" from background noise).

Parameters:

Name	Type	Description	Default
n_samples	int	Number of time samples (silhouettes) to composite. Defaults to 12.	12
method	str	Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.	'auto'
threshold	float	Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.	0.1
kernel_size	int	Morphological cleanup kernel for the silhouette (0 disables). Defaults to 5.	5
keep_largest	bool	Keep only the largest blob (the person). Defaults to False.	False
colorize	bool	Tint each silhouette by time (early→late) for a temporal cue. Defaults to True.	True
background	str	'average' (clean plate), 'first' (first frame), 'black' or 'white'. Defaults to 'average'.	'average'
target_name	str	Output name. Defaults to None ("_stroboscope.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	the stroboscope image.

Source code in musicalgestures/_spacetime.py

def mg_stroboscope(self, n_samples: int = 12, method: str = 'auto', threshold: float = 0.1, kernel_size: int = 5,
                   keep_largest: bool = False, colorize: bool = True, background: str = 'average',
                   target_name: str | None = None, overwrite: bool = True) -> "MgImage":
    """
    Renders a stroboscope / chronophotography image: the person's silhouette at evenly
    sampled times composited onto a single frame, showing the body moving through space
    over time (Muybridge-style).

    For a clean result with a single person on a static background, raise ``threshold`` and
    set ``keep_largest=True`` so only the person's blob is composited (avoids the image
    "blowing up" from background noise).

    Args:
        n_samples (int, optional): Number of time samples (silhouettes) to composite. Defaults to 12.
        method (str, optional): Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.
        threshold (float, optional): Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.
        kernel_size (int, optional): Morphological cleanup kernel for the silhouette (0 disables). Defaults to 5.
        keep_largest (bool, optional): Keep only the largest blob (the person). Defaults to False.
        colorize (bool, optional): Tint each silhouette by time (early→late) for a temporal cue. Defaults to True.
        background (str, optional): 'average' (clean plate), 'first' (first frame), 'black' or 'white'. Defaults to 'average'.
        target_name (str, optional): Output name. Defaults to None ("_stroboscope.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgImage: the stroboscope image.
    """
    target_name = resolve_filename(self.of, '_stroboscope.png', target_name, overwrite)

    avg = _average_frame(self)
    bg_gray = cv2.cvtColor(avg, cv2.COLOR_BGR2GRAY).astype(np.float32)
    seg_fn = _make_segmenter(method)

    total = int(self.length)
    sample_idx = set(np.linspace(0, total - 1, min(n_samples, total)).astype(int).tolist())

    if background == 'average':
        canvas = avg.copy()
    elif background == 'white':
        canvas = np.full((self.height, self.width, 3), 255, np.uint8)
    elif background == 'black':
        canvas = np.zeros((self.height, self.width, 3), np.uint8)
    else:
        canvas = None  # 'first' → set on first read

    import matplotlib
    cmap = matplotlib.colormaps['viridis']

    pb = MgProgressbar(total=self.length, prefix='Rendering stroboscope:')
    i = 0
    order = 0
    n_order = max(len(sample_idx) - 1, 1)
    for frame in _iter_frames(self):
        if canvas is None:
            canvas = frame.copy()
        if i in sample_idx:
            mask = _silhouette(frame, seg_fn, bg_gray, threshold, kernel_size, keep_largest)
            if colorize:
                tint = (np.array(cmap(order / n_order)[:3]) * 255)[::-1]  # RGB→BGR
                tinted = (frame.astype(np.float32) * 0.5 + tint * 0.5).astype(np.uint8)
                canvas[mask] = tinted[mask]
            else:
                canvas[mask] = frame[mask]
            order += 1
        i += 1
        pb.progress(i)
    pb.progress(self.length)

    cv2.imwrite(target_name, canvas)
    self.stroboscope_image = MgImage(target_name)
    return self.stroboscope_image

silhouette_waterfall

silhouette_waterfall(n_samples=40, method='auto', threshold=0.1, kernel_size=5, keep_largest=False, axis='horizontal', cmap='viridis', dpi=200, elev=35, azim=-60, axes=True, crop=False, target_name=None, overwrite=True)

Renders a 3D silhouette waterfall: the per-frame silhouette projected onto one spatial axis and stacked as cascading curves along a time (depth) axis, so the body's occupancy profile "flows" through time — like a 3D spectrogram waterfall.

For a single person on a static background, raise threshold and/or set keep_largest=True for a cleaner profile.

Parameters:

Name	Type	Description	Default
n_samples	int	Number of time slices (profiles) to stack. Defaults to 40.	40
method	str	Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.	'auto'
threshold	float	Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.	0.1
kernel_size	int	Morphological cleanup kernel (0 disables). Defaults to 5.	5
keep_largest	bool	Keep only the largest blob (the person). Defaults to False.	False
axis	str	'horizontal' profiles over x (collapse y); 'vertical' profiles over y. Defaults to 'horizontal'.	'horizontal'
cmap	str	Matplotlib colormap (by time). Defaults to 'viridis'.	'viridis'
dpi	int	Output DPI. Defaults to 200.	200
elev	float	3D elevation angle. Defaults to 35.	35
azim	float	3D azimuth angle. Defaults to -60.	-60
axes	bool	Draw the axes, tick labels, and title. Set to False for a clean render with all axes and text removed. Defaults to True.	True
crop	bool	Tighten the spatial axis to the occupied (nonzero) extent and trim the surrounding whitespace, so the figure shows mostly the data. Defaults to False.	False
target_name	str	Output name. Defaults to None ("_silhouette_waterfall.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	the 3D waterfall figure (the stacked profiles are in .data).

Source code in musicalgestures/_spacetime.py

def mg_silhouette_waterfall(self, n_samples: int = 40, method: str = 'auto', threshold: float = 0.1, kernel_size: int = 5,
                            keep_largest: bool = False, axis: str = 'horizontal', cmap: str = 'viridis', dpi: int = 200,
                            elev: float = 35, azim: float = -60, axes: bool = True, crop: bool = False, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Renders a 3D silhouette waterfall: the per-frame silhouette projected onto one spatial
    axis and stacked as cascading curves along a time (depth) axis, so the body's occupancy
    profile "flows" through time — like a 3D spectrogram waterfall.

    For a single person on a static background, raise ``threshold`` and/or set
    ``keep_largest=True`` for a cleaner profile.

    Args:
        n_samples (int, optional): Number of time slices (profiles) to stack. Defaults to 40.
        method (str, optional): Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.
        threshold (float, optional): Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.
        kernel_size (int, optional): Morphological cleanup kernel (0 disables). Defaults to 5.
        keep_largest (bool, optional): Keep only the largest blob (the person). Defaults to False.
        axis (str, optional): 'horizontal' profiles over x (collapse y); 'vertical' profiles over y. Defaults to 'horizontal'.
        cmap (str, optional): Matplotlib colormap (by time). Defaults to 'viridis'.
        dpi (int, optional): Output DPI. Defaults to 200.
        elev (float, optional): 3D elevation angle. Defaults to 35.
        azim (float, optional): 3D azimuth angle. Defaults to -60.
        axes (bool, optional): Draw the axes, tick labels, and title. Set to False for a clean
            render with all axes and text removed. Defaults to True.
        crop (bool, optional): Tighten the spatial axis to the occupied (nonzero) extent and trim
            the surrounding whitespace, so the figure shows mostly the data. Defaults to False.
        target_name (str, optional): Output name. Defaults to None ("_silhouette_waterfall.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgFigure: the 3D waterfall figure (the stacked profiles are in ``.data``).
    """
    import matplotlib
    import matplotlib.pyplot as plt
    from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 (registers 3d projection)

    target_name = resolve_filename(self.of, '_silhouette_waterfall.png', target_name, overwrite)

    avg = _average_frame(self)
    bg_gray = cv2.cvtColor(avg, cv2.COLOR_BGR2GRAY).astype(np.float32)
    seg_fn = _make_segmenter(method)

    total = int(self.length)
    sample_idx = sorted(set(np.linspace(0, total - 1, min(n_samples, total)).astype(int).tolist()))
    sample_set = set(sample_idx)

    profiles = []
    times = []
    pb = MgProgressbar(total=self.length, prefix='Rendering silhouette waterfall:')
    i = 0
    for frame in _iter_frames(self):
        if i in sample_set:
            mask = _silhouette(frame, seg_fn, bg_gray, threshold, kernel_size, keep_largest)
            if axis == 'horizontal':
                profiles.append(mask.sum(axis=0).astype(np.float32))   # over x (length W)
            else:
                profiles.append(mask.sum(axis=1).astype(np.float32))   # over y (length H)
            times.append(i / max(self.fps, 1))
        i += 1
        pb.progress(i)
    pb.progress(self.length)

    arr = np.array(profiles, dtype=np.float32)  # (n_slices, axis_len)
    if arr.size and arr.max() > 0:
        arr = arr / arr.max()

    cmap_obj = matplotlib.colormaps[cmap]
    fig = plt.figure(figsize=(11, 8), dpi=dpi)
    ax = fig.add_subplot(111, projection='3d')
    fig.patch.set_facecolor('white')

    pos = np.arange(arr.shape[1]) if arr.size else np.array([])
    n_slices = max(len(profiles) - 1, 1)
    for k, (prof, t) in enumerate(zip(arr, times)):
        ax.plot(pos, np.full_like(pos, t, dtype=float), prof,
                color=cmap_obj(k / n_slices), lw=0.9, alpha=0.9)

    if crop and arr.size:
        # Tighten the spatial axis to the occupied (nonzero) profile extent.
        occupied = np.where(arr.max(axis=0) > 0)[0]
        if occupied.size:
            pad = max(int((occupied[-1] - occupied[0]) * 0.05), 1)
            ax.set_xlim(max(occupied[0] - pad, 0), min(occupied[-1] + pad, arr.shape[1] - 1))

    if axes:
        ax.set_xlabel('Horizontal position (px)' if axis == 'horizontal' else 'Vertical position (px)')
        ax.set_ylabel('Time (s)')
        ax.set_zlabel('Silhouette extent')
        ax.set_title('Silhouette waterfall')
    else:
        ax.set_axis_off()
    ax.view_init(elev=elev, azim=azim)
    if crop:
        ax.set_position([0, 0, 1, 1])
        try:
            ax.set_box_aspect(None, zoom=1.5)
        except TypeError:
            pass
        save_kwargs = {'bbox_inches': 'tight', 'pad_inches': 0}
    else:
        fig.tight_layout()
        save_kwargs = {}
    fig.savefig(target_name, facecolor='white', **save_kwargs)
    plt.close(fig)

    data = {'profiles': arr, 'times': np.array(times), 'axis': axis}
    mgf = MgFigure(figure=fig, figure_type='video.silhouette_waterfall', data=data, layers=None, image=target_name)
    self.silhouette_waterfall_figure = mgf
    return mgf

motionhistory

motionhistory(threshold=0.05, decay=0.3, normalize=False, blur=0, cmap='hot', dpi=300, target_name=None, overwrite=True)

Renders a Motion History Image (Bobick & Davis): a single image where intensity encodes how recently motion occurred at each pixel (recent motion bright, older motion fades out).

A motion mark is set to full intensity where motion occurs and then decays linearly to zero over a window set by decay, so old motion disappears instead of accumulating and washing out the image. Raise threshold to ignore background noise, and lower decay for shorter (less crowded) trails.

Parameters:

Name	Type	Description	Default
threshold	float	Motion threshold (0–1) on frame differences. Higher rejects more background noise. Defaults to 0.05.	0.05
decay	float	Fade window as a fraction of the clip length (0–1): a motion mark fully fades after this fraction of the video. Smaller = shorter trails, less blow-out. Defaults to 0.3.	0.3
normalize	bool	Stretch the result to the full intensity range. Defaults to False. The MHI is already built in [0, 1], so normalization is rarely needed; when the final frames are static it amplifies faint residual trails and over-brightens ("blows up") the image, so it is guarded to skip when the peak intensity is very low.	False
blur	int	Optional Gaussian smoothing radius for the difference mask (0 = off). Helps suppress speckle noise. Defaults to 0.	0
cmap	str	Matplotlib colormap. Defaults to 'hot'.	'hot'
dpi	int	Output DPI. Defaults to 300.	300
target_name	str	Output name. Defaults to None ("_mhi.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgImage	'MgImage'	the motion history image.

Source code in musicalgestures/_spacetime.py

def mg_motionhistory(self, threshold: float = 0.05, decay: float = 0.3, normalize: bool = False, blur: int = 0,
                     cmap: str = 'hot', dpi: int = 300, target_name: str | None = None, overwrite: bool = True) -> "MgImage":
    """
    Renders a Motion History Image (Bobick & Davis): a single image where intensity encodes
    how recently motion occurred at each pixel (recent motion bright, older motion fades out).

    A motion mark is set to full intensity where motion occurs and then **decays** linearly to
    zero over a window set by ``decay``, so old motion disappears instead of accumulating and
    washing out the image. Raise ``threshold`` to ignore background noise, and lower ``decay``
    for shorter (less crowded) trails.

    Args:
        threshold (float, optional): Motion threshold (0–1) on frame differences. Higher rejects
            more background noise. Defaults to 0.05.
        decay (float, optional): Fade window as a fraction of the clip length (0–1): a motion
            mark fully fades after this fraction of the video. Smaller = shorter trails, less
            blow-out. Defaults to 0.3.
        normalize (bool, optional): Stretch the result to the full intensity range. Defaults to False.
            The MHI is already built in [0, 1], so normalization is rarely needed; when the final
            frames are static it amplifies faint residual trails and over-brightens ("blows up") the
            image, so it is guarded to skip when the peak intensity is very low.
        blur (int, optional): Optional Gaussian smoothing radius for the difference mask (0 = off).
            Helps suppress speckle noise. Defaults to 0.
        cmap (str, optional): Matplotlib colormap. Defaults to 'hot'.
        dpi (int, optional): Output DPI. Defaults to 300.
        target_name (str, optional): Output name. Defaults to None ("_mhi.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgImage: the motion history image.
    """
    import matplotlib
    import matplotlib.pyplot as plt

    target_name = resolve_filename(self.of, '_mhi.png', target_name, overwrite)

    total = max(int(self.length), 1)
    decay_frames = max(1, int(decay * total))
    step = 1.0 / decay_frames

    mhi = np.zeros((self.height, self.width), dtype=np.float32)
    prev_gray = None
    pb = MgProgressbar(total=self.length, prefix='Rendering motion history image:')
    i = 0
    for frame in _iter_frames(self):
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32)
        if prev_gray is not None:
            diff = np.abs(gray - prev_gray)
            if blur and blur > 0:
                k = int(blur) * 2 + 1
                diff = cv2.GaussianBlur(diff, (k, k), 0)
            motion = diff > (threshold * 255)
            # Decay everything, then re-stamp current motion to full intensity
            mhi -= step
            np.clip(mhi, 0.0, 1.0, out=mhi)
            mhi[motion] = 1.0
        prev_gray = gray
        i += 1
        pb.progress(i)
    pb.progress(self.length)

    # Only normalize when there is substantial motion intensity; otherwise dividing by a tiny
    # peak amplifies faint residual trails into a washed-out ("blown up") image.
    if normalize and mhi.max() > 0.2:
        mhi = mhi / mhi.max()

    fig, ax = plt.subplots(figsize=(12, 12 * self.height / self.width), dpi=dpi)
    fig.patch.set_facecolor('white')
    ax.imshow(mhi, cmap=cmap, vmin=0.0, vmax=1.0)
    ax.set_title('Motion History Image (bright = recent motion)')
    ax.axis('off')
    fig.tight_layout()
    fig.savefig(target_name, facecolor='white', bbox_inches='tight')
    plt.close(fig)

    self.mhi_image = MgImage(target_name)
    return self.mhi_image

spacetime_volume

spacetime_volume(n_samples=50, downsample=8, method='auto', threshold=0.1, kernel_size=5, keep_largest=False, cmap='viridis', dpi=200, elev=20, azim=-60, target_name=None, overwrite=True)

Renders a 3D space-time scatter of the person's silhouette: points (x, y, t) where the silhouette is present, with time on the depth axis and colour, showing how the body occupies space through time.

Parameters:

Name	Type	Description	Default
n_samples	int	Number of time samples (depth slices). Defaults to 50.	50
downsample	int	Spatial downsampling factor for the silhouette points. Defaults to 8.	8
method	str	Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.	'auto'
threshold	float	Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.	0.1
kernel_size	int	Morphological cleanup kernel for the silhouette (0 disables). Defaults to 5.	5
keep_largest	bool	Keep only the largest blob (the person). Defaults to False.	False
cmap	str	Matplotlib colormap for time. Defaults to 'viridis'.	'viridis'
dpi	int	Output DPI. Defaults to 200.	200
elev	float	3D elevation angle. Defaults to 20.	20
azim	float	3D azimuth angle. Defaults to -60.	-60
target_name	str	Output name. Defaults to None ("_spacetime_volume.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	the 3D space-time figure (data holds the point cloud).

Source code in musicalgestures/_spacetime.py

def mg_spacetime_volume(self, n_samples: int = 50, downsample: int = 8, method: str = 'auto', threshold: float = 0.1,
                        kernel_size: int = 5, keep_largest: bool = False, cmap: str = 'viridis', dpi: int = 200,
                        elev: float = 20, azim: float = -60, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Renders a 3D space-time scatter of the person's silhouette: points (x, y, t) where the
    silhouette is present, with time on the depth axis and colour, showing how the body
    occupies space through time.

    Args:
        n_samples (int, optional): Number of time samples (depth slices). Defaults to 50.
        downsample (int, optional): Spatial downsampling factor for the silhouette points. Defaults to 8.
        method (str, optional): Silhouette extraction: 'auto', 'mediapipe', or 'bgsub'. Defaults to 'auto'.
        threshold (float, optional): Foreground threshold (0–1). Higher rejects more background. Defaults to 0.1.
        kernel_size (int, optional): Morphological cleanup kernel for the silhouette (0 disables). Defaults to 5.
        keep_largest (bool, optional): Keep only the largest blob (the person). Defaults to False.
        cmap (str, optional): Matplotlib colormap for time. Defaults to 'viridis'.
        dpi (int, optional): Output DPI. Defaults to 200.
        elev (float, optional): 3D elevation angle. Defaults to 20.
        azim (float, optional): 3D azimuth angle. Defaults to -60.
        target_name (str, optional): Output name. Defaults to None ("_spacetime_volume.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgFigure: the 3D space-time figure (data holds the point cloud).
    """
    import matplotlib
    import matplotlib.pyplot as plt
    from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 (registers 3d projection)

    target_name = resolve_filename(self.of, '_spacetime_volume.png', target_name, overwrite)

    avg = _average_frame(self)
    bg_gray = cv2.cvtColor(avg, cv2.COLOR_BGR2GRAY).astype(np.float32)
    seg_fn = _make_segmenter(method)

    total = int(self.length)
    sample_idx = set(np.linspace(0, total - 1, min(n_samples, total)).astype(int).tolist())

    xs, ys, ts = [], [], []
    pb = MgProgressbar(total=self.length, prefix='Building space-time volume:')
    i = 0
    for frame in _iter_frames(self):
        if i in sample_idx:
            mask = _silhouette(frame, seg_fn, bg_gray, threshold, kernel_size, keep_largest)
            sub = mask[::downsample, ::downsample]
            yy, xx = np.nonzero(sub)
            xs.append(xx * downsample)
            ys.append(yy * downsample)
            ts.append(np.full(len(xx), i / max(self.fps, 1)))  # seconds
        i += 1
        pb.progress(i)
    pb.progress(self.length)

    xs = np.concatenate(xs) if xs else np.array([])
    ys = np.concatenate(ys) if ys else np.array([])
    ts = np.concatenate(ts) if ts else np.array([])

    fig = plt.figure(figsize=(10, 8), dpi=dpi)
    ax = fig.add_subplot(111, projection='3d')
    fig.patch.set_facecolor('white')
    if len(xs):
        ax.scatter(xs, ts, self.height - ys, c=ts, cmap=cmap, s=2, alpha=0.5, depthshade=True)
    ax.set_xlabel('x (px)')
    ax.set_ylabel('time (s)')
    ax.set_zlabel('y (px)')
    ax.set_title('Space-time silhouette volume')
    ax.view_init(elev=elev, azim=azim)
    fig.tight_layout()
    fig.savefig(target_name, facecolor='white')
    plt.close(fig)

    data = {'x': xs, 'y': ys, 't': ts, 'fps': self.fps}
    mgf = MgFigure(figure=fig, figure_type='video.spacetime_volume', data=data, layers=None, image=target_name)
    self.spacetime_volume_figure = mgf
    return mgf

beat_statistics

beat_statistics(source='motion', n_bins=32, cmap='YlOrRd', dpi=300, autoshow=True, title=None, target_name=None, overwrite=True, fmin=0.2, fmax=8.0)

Circular statistics of beat-timing consistency, from the audio or from the movement.

Fits an ideal isochronous beat grid to the detected beats and visualises how each beat deviates from it (a polar phase histogram with the mean resultant vector, plus a millisecond-deviation time series), revealing whether the rhythm rushes, drags, or stays steady. Requires at least four detected beats.

Parameters:

Name	Type	Description	Default
source	str	'motion' (default) detects rhythmic onsets in the video's quantity of motion and analyses the movement rhythm; 'audio' analyses the audio track instead (same as MgAudio.beat_statistics / video.audio.beat_statistics).	'motion'
n_bins	int	Bins in the polar phase histogram. Defaults to 32.	32
cmap	str	Colormap for the polar histogram. Defaults to 'YlOrRd'.	'YlOrRd'
dpi	int	Output DPI. Defaults to 300.	300
autoshow	bool	Kept for API parity (display is via show()). Defaults to True.	True
title	str	Optional figure title; use 'filename' for the file name. Defaults to None.	None
target_name	str	Output image name. Defaults to None.	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
fmin	float	Lowest movement-onset rate to consider (Hz), 'motion' only. Defaults to 0.2.	0.2
fmax	float	Highest movement-onset rate to consider (Hz), 'motion' only. Defaults to 8.0.	8.0

Returns:

Name	Type	Description
MgFigure	'MgFigure'	figure with the beat statistics in .data, or None if too few beats.

Source code in musicalgestures/_movementbeats.py

def mg_beat_statistics(self, source: str = 'motion', n_bins: int = 32, cmap: str = 'YlOrRd', dpi: int = 300,
                       autoshow: bool = True, title: str | None = None, target_name: str | None = None, overwrite: bool = True,
                       fmin: float = 0.2, fmax: float = 8.0) -> "MgFigure":
    """
    Circular statistics of beat-timing consistency, from the **audio** or from the **movement**.

    Fits an ideal isochronous beat grid to the detected beats and visualises how each beat
    deviates from it (a polar phase histogram with the mean resultant vector, plus a
    millisecond-deviation time series), revealing whether the rhythm rushes, drags, or stays
    steady. Requires at least four detected beats.

    Args:
        source (str, optional): `'motion'` (default) detects rhythmic onsets in the video's
            quantity of motion and analyses the **movement** rhythm; `'audio'` analyses the
            audio track instead (same as `MgAudio.beat_statistics` / `video.audio.beat_statistics`).
        n_bins (int, optional): Bins in the polar phase histogram. Defaults to 32.
        cmap (str, optional): Colormap for the polar histogram. Defaults to 'YlOrRd'.
        dpi (int, optional): Output DPI. Defaults to 300.
        autoshow (bool, optional): Kept for API parity (display is via show()). Defaults to True.
        title (str, optional): Optional figure title; use 'filename' for the file name. Defaults to None.
        target_name (str, optional): Output image name. Defaults to None.
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        fmin (float, optional): Lowest movement-onset rate to consider (Hz), 'motion' only. Defaults to 0.2.
        fmax (float, optional): Highest movement-onset rate to consider (Hz), 'motion' only. Defaults to 8.0.

    Returns:
        MgFigure: figure with the beat statistics in ``.data``, or None if too few beats.
    """
    source = str(source).lower()
    if source == 'audio':
        # Delegate to the inherited audio implementation (operates on the audio track)
        from musicalgestures._audio import MgAudio
        return MgAudio.beat_statistics(self, n_bins=n_bins, cmap=cmap, dpi=dpi,
                                       autoshow=autoshow, title=title,
                                       target_name=target_name, overwrite=overwrite)
    if source != 'motion':
        raise ValueError("source must be 'audio' or 'motion'.")

    from musicalgestures._analysis import circular_stats, rayleigh_test
    import librosa

    qom, fps = _movement_qom(self)
    if len(qom) < 8:
        print('Not enough frames to detect movement beats.')
        return

    # Detect movement onsets ("beats") as peaks in the quantity-of-motion envelope.
    beat_times = librosa.onset.onset_detect(
        onset_envelope=qom, sr=fps, hop_length=1, units='time', backtrack=False)
    # Keep onsets within the plausible movement-rate band via inter-onset interval
    if len(beat_times) >= 2:
        ibi_all = np.diff(beat_times)
        keep = (ibi_all >= 1.0 / fmax) & (ibi_all <= 1.0 / fmin)
        beat_times = np.concatenate([beat_times[:1], beat_times[1:][keep]])

    if len(beat_times) < 4:
        print('Not enough movement beats detected for circular statistics (need at least 4).')
        return

    # Circular grid statistics (same model as the audio version)
    k = np.arange(len(beat_times))
    T_fit, t0_fit = np.polyfit(k, beat_times, 1)
    deviations_s = beat_times - (t0_fit + k * T_fit)
    beat_phases = (deviations_s / T_fit) * 2 * np.pi % (2 * np.pi)
    R_beat, mu_beat = circular_stats(beat_phases)
    _, p_rayleigh = rayleigh_test(beat_phases)
    ibi = np.diff(beat_times)
    beat_regularity = float(1.0 - ibi.std() / ibi.mean()) if len(ibi) and ibi.mean() > 0 else 0.0
    tempo = 60.0 / T_fit if T_fit > 0 else 0.0

    d = {
        'source': 'motion',
        'fps': fps,
        'of': self.of,
        'tempo': tempo,
        'beat_times': beat_times,
        'ibi': ibi,
        'beat_regularity': beat_regularity,
        'beat_phases': beat_phases,
        'deviations_s': deviations_s,
        'R_beat': R_beat,
        'mu_beat': mu_beat,
        'T_fit': T_fit,
        't0_fit': t0_fit,
        'p_rayleigh': p_rayleigh,
        'qom': qom,
    }

    target_name = resolve_filename(self.of, '_movement_beatstats.png', target_name, overwrite)

    deviations_ms = deviations_s * 1000
    R, mu = R_beat, mu_beat

    fig = plt.figure(figsize=(14, 6), dpi=dpi)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    # Polar histogram of movement-beat phases
    ax_p = fig.add_subplot(121, projection='polar')
    bin_edges = np.linspace(0, 2 * np.pi, n_bins + 1)
    counts, _ = np.histogram(beat_phases, bins=bin_edges)
    theta_c = (bin_edges[:-1] + bin_edges[1:]) / 2
    norm_c = counts / (counts.max() + 1e-9)
    ax_p.bar(theta_c, counts, width=2 * np.pi / n_bins * 0.88,
             color=matplotlib.colormaps[cmap](norm_c), alpha=0.85,
             edgecolor='white', linewidth=0.3)
    if counts.max() > 0:
        ax_p.annotate('', xy=(np.radians(mu), R * counts.max()), xytext=(0, 0),
                      arrowprops=dict(arrowstyle='-|>', color='#333333', lw=2.0, mutation_scale=16))
    ax_p.set_xticks([0, np.pi / 2, np.pi, 3 * np.pi / 2])
    ax_p.set_xticklabels(['on beat', '1/4 late', '1/2', '1/4 early'], fontsize=8)
    ax_p.set_title(f'Movement-beat phase deviation\nR = {R:.3f}   μ = {mu:.1f}°   p = {p_rayleigh:.4f}', fontsize=10)

    # Time series of deviations
    ax_t = fig.add_subplot(122)
    sc = ax_t.scatter(beat_times, deviations_ms, c=beat_times, cmap='plasma', s=18, alpha=0.8)
    ax_t.axhline(0, color='#888888', lw=1.0, ls='--', alpha=0.7)
    ax_t.axhline(float(deviations_ms.mean()), color='#1f77b4', lw=1.2, ls=':',
                 label=f'mean {float(deviations_ms.mean()):.1f} ms')
    ax_t.set(xlabel='Time (s)', ylabel='Deviation from ideal grid (ms)', title='Movement-beat timing deviation')
    ax_t.legend()
    cb = fig.colorbar(sc, ax=ax_t)
    cb.set_label('Time (s)')

    if title is None:
        title = ''
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(f'{title}   Movement tempo: {tempo:.1f} BPM   σ = {float(deviations_ms.std()):.1f} ms'.strip(),
                 fontsize=13, fontweight='bold')

    plt.tight_layout(rect=[0, 0, 1, 0.93])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    mgf = MgFigure(figure=fig, figure_type='video.beat_statistics', data=d, layers=None, image=target_name)
    self.movement_beat_statistics = mgf
    return mgf

tempo_similarity

tempo_similarity(dpi=300, autoshow=True, title=None, target_name=None, overwrite=True)

Compare the audio tempo/rhythm with the movement tempo/rhythm and report how similar they are.

Estimates the tempo of the audio track (from its onset-strength envelope) and of the movement (from the quantity-of-motion envelope), then aligns the two normalised envelopes and cross-correlates them to measure rhythmic agreement. The figure shows the two envelopes overlaid and their cross-correlation; the report (also saved as a CSV) lists the audio tempo, movement tempo, their ratio and nearest harmonic relationship, the peak cross-correlation, and the lag (s) at which the movement best aligns with the audio.

Parameters:

Name	Type	Description	Default
dpi	int	Output DPI. Defaults to 300.	300
autoshow	bool	Kept for API parity (display via show()). Defaults to True.	True
title	str	Optional figure title; 'filename' uses the file name. Defaults to None.	None
target_name	str	Output image name. Defaults to None ("_tempo_similarity.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	the report figure (metrics in .data), or None if the video has no audio.

Source code in musicalgestures/_movementbeats.py

def mg_tempo_similarity(self, dpi: int = 300, autoshow: bool = True, title: str | None = None, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Compare the **audio** tempo/rhythm with the **movement** tempo/rhythm and report how similar
    they are.

    Estimates the tempo of the audio track (from its onset-strength envelope) and of the movement
    (from the quantity-of-motion envelope), then aligns the two normalised envelopes and
    cross-correlates them to measure rhythmic agreement. The figure shows the two envelopes
    overlaid and their cross-correlation; the report (also saved as a CSV) lists the audio tempo,
    movement tempo, their ratio and nearest harmonic relationship, the peak cross-correlation, and
    the lag (s) at which the movement best aligns with the audio.

    Args:
        dpi (int, optional): Output DPI. Defaults to 300.
        autoshow (bool, optional): Kept for API parity (display via show()). Defaults to True.
        title (str, optional): Optional figure title; 'filename' uses the file name. Defaults to None.
        target_name (str, optional): Output image name. Defaults to None ("_tempo_similarity.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgFigure: the report figure (metrics in ``.data``), or None if the video has no audio.
    """
    import librosa
    from musicalgestures._utils import has_audio

    if not has_audio(self.filename):
        print('The video has no audio track — cannot compare audio and movement tempo.')
        return None

    target_name = resolve_filename(self.of, '_tempo_similarity.png', target_name, overwrite)

    # --- Audio envelope + tempo ---
    y, sr = self._load()
    hop = self.hop_length
    oenv = librosa.onset.onset_strength(y=y, sr=sr, hop_length=hop)
    audio_tempo = float(np.atleast_1d(librosa.feature.tempo(onset_envelope=oenv, sr=sr, hop_length=hop))[0])
    a_t = librosa.times_like(oenv, sr=sr, hop_length=hop)

    # --- Movement envelope + tempo ---
    qom, fps = _movement_qom(self)
    if len(qom) < 4:
        print('Not enough frames to estimate movement tempo.')
        return None
    motion_tempo = float(np.atleast_1d(librosa.feature.tempo(onset_envelope=qom, sr=fps, hop_length=1))[0])
    m_t = np.arange(len(qom)) / max(fps, 1e-9)

    # --- Resample both onto a common time grid and cross-correlate ---
    fs = 50.0  # Hz
    dur = min(a_t[-1] if len(a_t) else 0, m_t[-1] if len(m_t) else 0)
    if dur <= 0:
        print('Audio and movement do not overlap in time.')
        return None
    t = np.arange(0, dur, 1.0 / fs)
    a = np.interp(t, a_t, oenv)
    m = np.interp(t, m_t, qom)
    a = (a - a.mean()) / (a.std() + 1e-9)
    m = (m - m.mean()) / (m.std() + 1e-9)
    xcorr = np.correlate(m, a, mode='full') / len(t)
    lags = np.arange(-len(t) + 1, len(t)) / fs
    peak_i = int(np.argmax(xcorr))
    peak_lag = float(lags[peak_i])
    peak_corr = float(xcorr[peak_i])
    zero_corr = float(xcorr[len(t) - 1])  # correlation at zero lag

    ratio = motion_tempo / audio_tempo if audio_tempo > 0 else 0.0
    harm, harm_label = _nearest_harmonic_ratio(ratio) if ratio > 0 else (0.0, 'n/a')
    tempo_agreement = float(max(0.0, 1.0 - abs(np.log2(ratio / harm)))) if (ratio > 0 and harm > 0) else 0.0

    # --- Figure ---
    fig, axes = plt.subplots(2, 1, figsize=(12, 7), dpi=dpi)
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Audio–movement tempo similarity', fontsize=14)

    axes[0].plot(t, a, color='#1f77b4', lw=0.9, label=f'Audio onset (tempo {audio_tempo:.1f} BPM)')
    axes[0].plot(t, m, color='#d62728', lw=0.9, alpha=0.8, label=f'Movement QoM (tempo {motion_tempo:.1f} BPM)')
    axes[0].set_xlabel('Time (s)')
    axes[0].set_ylabel('Normalised envelope')
    axes[0].legend(loc='upper right', fontsize=8)
    axes[0].set_title('Audio onset strength vs. quantity of motion', fontsize=10)

    axes[1].plot(lags, xcorr, color='#2ca02c', lw=0.9)
    axes[1].axvline(peak_lag, color='crimson', ls='--', lw=1,
                    label=f'peak r={peak_corr:.2f} @ {peak_lag:+.2f}s')
    axes[1].axvline(0, color='#888888', ls=':', lw=0.8)
    axes[1].set_xlabel('Lag (s)  — movement relative to audio')
    axes[1].set_ylabel('Cross-correlation')
    axes[1].legend(loc='upper right', fontsize=8)
    axes[1].set_title(
        f'Tempo ratio {ratio:.2f} (≈ {harm_label})  |  agreement {tempo_agreement:.2f}  |  '
        f'zero-lag r={zero_corr:.2f}', fontsize=10)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    d = {
        'audio_tempo_bpm': round(audio_tempo, 2),
        'motion_tempo_bpm': round(motion_tempo, 2),
        'tempo_ratio': round(ratio, 3),
        'nearest_harmonic': harm_label,
        'tempo_agreement': round(tempo_agreement, 3),
        'peak_crosscorr': round(peak_corr, 3),
        'peak_lag_s': round(peak_lag, 3),
        'zero_lag_crosscorr': round(zero_corr, 3),
        'fps': fps, 'sr': sr,
    }
    try:
        import pandas as pd
        pd.DataFrame([d]).to_csv(os.path.splitext(target_name)[0] + '.csv', index=False)
    except Exception as e:
        print(f'Warning: could not save CSV: {e}')

    mgf = MgFigure(figure=fig, figure_type='video.tempo_similarity', data=d, layers=None, image=target_name)
    self.tempo_similarity_figure = mgf
    return mgf

motiondescriptors

motiondescriptors(window='hann', entropy_bins=50, fmin=0.2, fmax=10.0, save_data=True, save_plot=True, data_format='csv', target_name=None, overwrite=True)

Scalar movement descriptors derived from the quantity-of-motion (QoM) signal.

Computes a compact set of higher-level descriptors that summarise how something moves, complementing the per-frame motion data from :func:motion:

• motion_energy — mean squared QoM; the overall amount of movement.
• motion_smoothness — SPARC (spectral arc length) of the QoM profile; a dimensionless, validated smoothness metric (less negative = smoother, more negative = jerkier).
• motion_entropy — normalised (0–1) Shannon entropy of the QoM magnitude distribution; the complexity/variedness of the motion.
• spectral descriptors of the QoM signal (Hann-windowed by default): the dominant frequency (Hz, the main movement-rhythm rate) and the spectral centroid (Hz, the "centre of mass" of the movement spectrum).

Parameters:

Name	Type	Description	Default
window	str	FFT window for the spectral descriptors — 'hann' (default, recommended to reduce leakage) or 'none' for a rectangular window.	'hann'
entropy_bins	int	Number of histogram bins for the entropy estimate. Defaults to 50.	50
fmin	float	Lowest frequency (Hz) considered for the dominant frequency and spectral centroid, excluding slow amplitude drift near DC. Defaults to 0.2.	0.2
fmax	float	Highest frequency (Hz) considered for those spectral descriptors. Defaults to 10.0.	10.0
save_data	bool	Save the descriptors to a data file. Defaults to True.	True
save_plot	bool	Save the figure (QoM time series + power spectrum). Defaults to True.	True
data_format	str	Data file format: 'csv', 'tsv' or 'txt'. Defaults to 'csv'.	'csv'
target_name	str	Output image name. Defaults to None (<name>_motiondescriptors.png).	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True

Returns:

Name	Type	Description
MgFigure	'MgFigure'	figure whose .data holds the scalar descriptors and the spectrum arrays
	'MgFigure'	(frequencies, power), or None if there are too few frames.

Source code in musicalgestures/_motiondescriptors.py

def mg_motiondescriptors(self, window: str = 'hann', entropy_bins: int = 50,
                         fmin: float = 0.2, fmax: float = 10.0,
                         save_data: bool = True, save_plot: bool = True,
                         data_format: str = 'csv', target_name: str | None = None,
                         overwrite: bool = True) -> "MgFigure":
    """Scalar movement descriptors derived from the quantity-of-motion (QoM) signal.

    Computes a compact set of higher-level descriptors that summarise *how* something moves,
    complementing the per-frame motion data from :func:`motion`:

    - **motion_energy** — mean squared QoM; the overall amount of movement.
    - **motion_smoothness** — SPARC (spectral arc length) of the QoM profile; a dimensionless,
      validated smoothness metric (less negative = smoother, more negative = jerkier).
    - **motion_entropy** — normalised (0–1) Shannon entropy of the QoM magnitude distribution;
      the complexity/variedness of the motion.
    - **spectral descriptors** of the QoM signal (Hann-windowed by default): the **dominant
      frequency** (Hz, the main movement-rhythm rate) and the **spectral centroid** (Hz, the
      "centre of mass" of the movement spectrum).

    Args:
        window (str, optional): FFT window for the spectral descriptors — 'hann' (default,
            recommended to reduce leakage) or 'none' for a rectangular window.
        entropy_bins (int, optional): Number of histogram bins for the entropy estimate. Defaults to 50.
        fmin (float, optional): Lowest frequency (Hz) considered for the dominant frequency and
            spectral centroid, excluding slow amplitude drift near DC. Defaults to 0.2.
        fmax (float, optional): Highest frequency (Hz) considered for those spectral descriptors.
            Defaults to 10.0.
        save_data (bool, optional): Save the descriptors to a data file. Defaults to True.
        save_plot (bool, optional): Save the figure (QoM time series + power spectrum). Defaults to True.
        data_format (str, optional): Data file format: 'csv', 'tsv' or 'txt'. Defaults to 'csv'.
        target_name (str, optional): Output image name. Defaults to None (``<name>_motiondescriptors.png``).
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.

    Returns:
        MgFigure: figure whose ``.data`` holds the scalar descriptors and the spectrum arrays
        (``frequencies``, ``power``), or None if there are too few frames.
    """
    qom, fps = _movement_qom(self)
    if qom.size < 4:
        print('Not enough frames to compute motion descriptors.')
        return None

    motion_energy = float(np.mean(qom ** 2))
    motion_smoothness = _sparc(qom, fps)
    motion_entropy = _motion_entropy(qom, bins=entropy_bins)
    freqs, power = _qom_spectrum(qom, fps, window=window)

    # Restrict the spectral descriptors to a movement band so slow amplitude drift near DC
    # doesn't masquerade as the dominant movement rhythm.
    band = (freqs >= fmin) & (freqs <= fmax)
    if band.any() and power[band].any():
        f_band, p_band = freqs[band], power[band]
        dominant_freq = float(f_band[int(np.argmax(p_band))])
        spectral_centroid = float(np.sum(f_band * p_band) / np.sum(p_band))
    else:
        dominant_freq = 0.0
        spectral_centroid = 0.0

    d = {
        'of': self.of,
        'fps': fps,
        'n_frames': int(qom.size),
        'motion_energy': motion_energy,
        'motion_smoothness': motion_smoothness,
        'motion_entropy': motion_entropy,
        'dominant_freq': dominant_freq,
        'spectral_centroid': spectral_centroid,
        'window': window,
        'fmin': fmin,
        'fmax': fmax,
        'qom': qom,
        'frequencies': freqs,
        'power': power,
    }

    if save_data:
        _save_descriptors(self.of, d, data_format, overwrite)

    target_name = resolve_filename(self.of, '_motiondescriptors.png', target_name, overwrite)

    times = np.arange(qom.size) / fps
    fig, (ax_q, ax_s) = plt.subplots(2, 1, figsize=(12, 7), dpi=300)
    fig.patch.set_facecolor('white')
    fig.patch.set_alpha(1)

    ax_q.plot(times, qom, color='#1f77b4', lw=1.0)
    ax_q.fill_between(times, qom, color='#1f77b4', alpha=0.15)
    ax_q.set(xlabel='Time (s)', ylabel='Quantity of motion', title='Quantity of motion over time')
    ax_q.margins(x=0)

    # Power spectrum up to a sensible movement band (10 Hz), dominant frequency marked.
    band = freqs <= 10.0 if freqs.size else slice(None)
    ax_s.plot(freqs[band], power[band], color='#d62728', lw=1.0)
    ax_s.fill_between(freqs[band], power[band], color='#d62728', alpha=0.15)
    if dominant_freq > 0:
        ax_s.axvline(dominant_freq, color='#333333', ls='--', lw=1.0,
                     label=f'dominant {dominant_freq:.2f} Hz')
        ax_s.legend()
    ax_s.set(xlabel='Frequency (Hz)', ylabel='Power',
             title=f'QoM power spectrum ({window} window)')
    ax_s.margins(x=0)

    summary = (f'energy = {motion_energy:.3g}    smoothness (SPARC) = {motion_smoothness:.3f}    '
               f'entropy = {motion_entropy:.3f}    spectral centroid = {spectral_centroid:.2f} Hz')
    fig.suptitle(summary, fontsize=11, fontweight='bold')

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    if save_plot:
        plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    mgf = MgFigure(figure=fig, figure_type='video.motiondescriptors', data=d, layers=None,
                   image=target_name if save_plot else None)
    self.motiondescriptors_figure = mgf
    return mgf

phase_synchrony

phase_synchrony(fmin=0.5, fmax=4.0, fs=50.0, n_bins=36, dpi=300, autoshow=True, title=None, target_name=None, overwrite=True)

Quantify how phase-locked the movement is to the audio rhythm.

Both the audio onset-strength envelope and the movement quantity-of-motion envelope are band-pass filtered to the tempo band [fmin, fmax] Hz, and their instantaneous phases (via the Hilbert transform) are compared. The phase-locking value (PLV, 0–1) summarises the consistency of the audio↔movement phase difference; a polar histogram shows its distribution.

Returns an MgFigure (metrics in .data), or None if the video has no audio.

Source code in musicalgestures/_audio_video.py

def mg_phase_synchrony(self, fmin: float = 0.5, fmax: float = 4.0, fs: float = 50.0, n_bins: int = 36, dpi: int = 300,
                       autoshow: bool = True, title: str | None = None, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Quantify how phase-locked the movement is to the audio rhythm.

    Both the audio onset-strength envelope and the movement quantity-of-motion envelope are
    band-pass filtered to the tempo band [``fmin``, ``fmax``] Hz, and their instantaneous phases
    (via the Hilbert transform) are compared. The **phase-locking value** (PLV, 0–1) summarises the
    consistency of the audio↔movement phase difference; a polar histogram shows its distribution.

    Returns an MgFigure (metrics in ``.data``), or None if the video has no audio.
    """
    from scipy.signal import butter, filtfilt, hilbert
    from musicalgestures._movementbeats import _movement_qom

    if not has_audio(self.filename):
        print('The video has no audio track.')
        return None

    target_name = resolve_filename(self.of, '_phase_synchrony.png', target_name, overwrite)

    oenv, t_a, sr = _audio_env(self, 'onset')
    qom, fps = _movement_qom(self)
    t_m = np.arange(len(qom)) / max(fps, 1e-9)
    t, a, m = _common_grid(t_a, oenv, t_m, qom, fs)
    if t is None or len(t) < 8:
        print('Audio and movement do not overlap enough in time.')
        return None

    ny = fs / 2.0
    b, aa = butter(2, [max(fmin, 0.01) / ny, min(fmax, ny - 0.01) / ny], btype='band')
    af = filtfilt(b, aa, _z(a))
    mf = filtfilt(b, aa, _z(m))
    pa = np.angle(hilbert(af))
    pm = np.angle(hilbert(mf))
    dphi = pm - pa
    z = np.exp(1j * dphi)
    plv = float(np.abs(np.mean(z)))
    mean_dphi = float(np.degrees(np.angle(np.mean(z))))

    fig = plt.figure(figsize=(12, 5), dpi=dpi)
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Audio–movement phase synchrony', fontsize=14)

    ax1 = fig.add_subplot(1, 2, 1)
    ax1.plot(t, af, color='#1f77b4', lw=0.8, label='Audio (band-passed)')
    ax1.plot(t, mf, color='#d62728', lw=0.8, alpha=0.8, label='Movement (band-passed)')
    ax1.set_xlabel('Time (s)')
    ax1.set_ylabel('Amplitude')
    ax1.legend(loc='upper right', fontsize=8)
    ax1.set_title(f'Tempo band {fmin}–{fmax} Hz', fontsize=10)

    ax2 = fig.add_subplot(1, 2, 2, projection='polar')
    counts, edges = np.histogram(dphi, bins=np.linspace(-np.pi, np.pi, n_bins + 1))
    centers = edges[:-1] + np.diff(edges) / 2
    cmax = counts.max() if counts.max() > 0 else 1
    ax2.bar(centers, counts, width=np.diff(edges), bottom=0.0,
            color=matplotlib.colormaps['viridis'](counts / cmax), alpha=0.85, edgecolor='none')
    ax2.plot([np.radians(mean_dphi), np.radians(mean_dphi)], [0, plv * cmax],
             color='crimson', lw=2, zorder=5)
    ax2.set_yticklabels([])
    ax2.set_title(f'Phase difference (movement − audio)\nPLV = {plv:.2f}, mean Δφ = {mean_dphi:.0f}°',
                  fontsize=10)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    d = {'plv': round(plv, 3), 'mean_phase_diff_deg': round(mean_dphi, 1),
         'fmin': fmin, 'fmax': fmax, 'fps': fps, 'sr': sr}
    mgf = MgFigure(figure=fig, figure_type='video.phase_synchrony', data=d, layers=None, image=target_name)
    self.phase_synchrony_figure = mgf
    return mgf

structure_comparison

structure_comparison(n=200, dpi=300, cmap='magma', autoshow=True, title=None, target_name=None, overwrite=True)

Compare the temporal structure of the audio with that of the movement.

Builds a self-similarity matrix (SSM) of the audio (from MFCC frames) and of the video (from low-resolution frame appearance), resampled to the same n time points, and shows them side by side with their absolute difference map — bright regions in the difference are where the musical structure and the movement structure diverge.

Returns an MgFigure (mean structural agreement in .data), or None if the video has no audio.

Source code in musicalgestures/_audio_video.py

def mg_structure_comparison(self, n: int = 200, dpi: int = 300, cmap: str = 'magma', autoshow: bool = True,
                            title: str | None = None, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Compare the temporal **structure** of the audio with that of the movement.

    Builds a self-similarity matrix (SSM) of the audio (from MFCC frames) and of the video
    (from low-resolution frame appearance), resampled to the same ``n`` time points, and shows
    them side by side with their absolute **difference map** — bright regions in the difference
    are where the musical structure and the movement structure diverge.

    Returns an MgFigure (mean structural agreement in ``.data``), or None if the video has no audio.
    """
    import librosa

    if not has_audio(self.filename):
        print('The video has no audio track.')
        return None

    target_name = resolve_filename(self.of, '_structure_comparison.png', target_name, overwrite)

    # Audio feature matrix (MFCC) → resample to n columns
    y, sr = self._load()
    mfcc = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=20, hop_length=self.hop_length)  # (20, Ta)
    idx_a = np.linspace(0, mfcc.shape[1] - 1, n).astype(int)
    audio_feat = mfcc[:, idx_a].T  # (n, 20)

    # Motion feature: low-res grayscale frame appearance at n evenly spaced frames
    cap = cv2.VideoCapture(self.filename)
    total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) or 1
    want = set(np.linspace(0, total - 1, min(n, total)).astype(int).tolist())
    feats, i = [], 0
    while True:
        ret, frame = cap.read()
        if not ret:
            break
        if i in want:
            small = cv2.resize(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY), (32, 32)).astype(np.float32)
            feats.append(small.ravel())
        i += 1
    cap.release()
    motion_feat = np.array(feats) if feats else np.zeros((n, 1024))
    # Match lengths (resample motion rows to n)
    if motion_feat.shape[0] != n and motion_feat.shape[0] > 1:
        ridx = np.linspace(0, motion_feat.shape[0] - 1, n).astype(int)
        motion_feat = motion_feat[ridx]

    ssm_audio = _ssm_from_features(audio_feat)
    ssm_motion = _ssm_from_features(motion_feat)
    k = min(ssm_audio.shape[0], ssm_motion.shape[0])
    ssm_audio, ssm_motion = ssm_audio[:k, :k], ssm_motion[:k, :k]
    diff = np.abs(ssm_audio - ssm_motion)
    agreement = float(1.0 - diff.mean())

    fig, axes = plt.subplots(1, 3, figsize=(15, 5), dpi=dpi)
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Audio vs movement structural similarity', fontsize=14)
    for ax, mat, ttl, cm in zip(axes, [ssm_audio, ssm_motion, diff],
                                ['Audio SSM (MFCC)', 'Movement SSM (appearance)',
                                 f'|difference|  (agreement {agreement:.2f})'],
                                [cmap, cmap, 'inferno']):
        im = ax.imshow(mat, origin='lower', cmap=cm, aspect='equal')
        ax.set_title(ttl, fontsize=10)
        ax.set_xticks([]); ax.set_yticks([])
        fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    d = {'structural_agreement': round(agreement, 3), 'n': int(k), 'sr': sr}
    mgf = MgFigure(figure=fig, figure_type='video.structure_comparison', data=d, layers=None, image=target_name)
    self.structure_comparison_figure = mgf
    return mgf

body_audio_coupling

body_audio_coupling(dpi=300, cmap='coolwarm', dot_size=260, autoshow=True, title=None, target_name=None, overwrite=True, **pose_kwargs)

Map which body parts are most rhythmically coupled to the music.

For every pose marker the per-frame speed is correlated with the audio onset-strength envelope (sampled at the video frame rate). The result is shown as a body map — the average pose with each marker coloured by its correlation — plus a sorted bar chart, and a CSV of the per-marker correlations. Uses cached pose keypoints when available, otherwise runs pose() first (**pose_kwargs are forwarded).

Returns an MgFigure (per-marker correlations in .data), or None if the video has no audio.

Source code in musicalgestures/_audio_video.py

def mg_body_audio_coupling(self, dpi: int = 300, cmap: str = 'coolwarm', dot_size: int = 260, autoshow: bool = True,
                           title: str | None = None, target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure":
    """
    Map which body parts are most rhythmically coupled to the music.

    For every pose marker the per-frame speed is correlated with the audio onset-strength
    envelope (sampled at the video frame rate). The result is shown as a body map — the average
    pose with each marker coloured by its correlation — plus a sorted bar chart, and a CSV of the
    per-marker correlations. Uses cached pose keypoints when available, otherwise runs ``pose()``
    first (``**pose_kwargs`` are forwarded).

    Returns an MgFigure (per-marker correlations in ``.data``), or None if the video has no audio.
    """
    from musicalgestures._pose_visualize import _positions_from_data

    if not has_audio(self.filename):
        print('The video has no audio track.')
        return None

    if getattr(self, '_pose_keypoints', None) is None:
        pose_kwargs.setdefault('save_video', False)
        pose_kwargs.setdefault('save_average_pose', False)
        pose_kwargs.setdefault('save_trajectories', False)
        self.pose(**pose_kwargs)
    c = self._pose_keypoints
    names, connections = c['names'], c.get('connections') or []
    width, height, fps = c['width'], c['height'], c['fps']

    target_name = resolve_filename(c['of'], '_body_audio_coupling.png', target_name, overwrite)

    coords, _ = _positions_from_data(c['data'], len(names))  # (T, n, 2) normalised
    px = coords * np.array([width, height])
    speed = np.sqrt((np.diff(px, axis=0) ** 2).sum(axis=2))   # (T-1, n) px/frame

    # Audio onset envelope sampled at the video frame times (length T), then aligned to speed.
    oenv, t_a, sr = _audio_env(self, 'onset')
    t_frames = np.arange(coords.shape[0]) / max(fps, 1e-9)
    aud_per_frame = np.interp(t_frames, t_a, oenv)[1:]  # align to diff length

    corrs = np.array([_safe_corr(np.nan_to_num(speed[:, i]), aud_per_frame) for i in range(len(names))])

    mean_px = np.nanmean(px, axis=0)
    vmax = float(np.nanmax(np.abs(corrs))) if np.isfinite(corrs).any() else 1.0
    vmax = max(vmax, 1e-6)
    cmap_obj = matplotlib.colormaps[cmap]
    norm = matplotlib.colors.Normalize(vmin=-vmax, vmax=vmax)

    fig, (axb, axbar) = plt.subplots(1, 2, figsize=(14, 7), dpi=dpi,
                                     gridspec_kw={'width_ratios': [1, 1]})
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Per-body-part coupling to the music', fontsize=14)

    # Body map
    for a, b in connections:
        if a < len(names) and b < len(names) and not (np.isnan(mean_px[a]).any() or np.isnan(mean_px[b]).any()):
            axb.plot([mean_px[a, 0], mean_px[b, 0]], [mean_px[a, 1], mean_px[b, 1]],
                     color='#bbbbbb', lw=2, zorder=1)
    for i in range(len(names)):
        if not np.isnan(mean_px[i]).any():
            axb.scatter(mean_px[i, 0], mean_px[i, 1], s=dot_size, c=[cmap_obj(norm(corrs[i]))],
                        edgecolors='black', linewidths=0.6, zorder=2)
    axb.set_xlim(0, width); axb.set_ylim(height, 0); axb.set_aspect('equal'); axb.axis('off')
    axb.set_title('Body map (marker colour = correlation of speed with audio)', fontsize=10)
    fig.colorbar(matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap_obj), ax=axb,
                 fraction=0.046, pad=0.04, label='Pearson r')

    # Sorted bar chart
    order = np.argsort(corrs)
    axbar.barh(np.arange(len(names)), corrs[order],
               color=[cmap_obj(norm(corrs[o])) for o in order])
    axbar.set_yticks(np.arange(len(names)))
    axbar.set_yticklabels([names[o] for o in order], fontsize=5)
    axbar.axvline(0, color='#888888', lw=0.8)
    axbar.set_xlabel('Correlation of marker speed with audio onset (Pearson r)')
    axbar.set_title('Per-marker coupling, ranked', fontsize=10)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    stats = [{'Marker': names[i], 'Correlation': round(float(corrs[i]), 3)} for i in range(len(names))]
    try:
        import pandas as pd
        pd.DataFrame(stats).to_csv(os.path.splitext(target_name)[0] + '.csv', index=False)
    except Exception as e:
        print(f'Warning: could not save CSV: {e}')

    d = {'correlations': stats, 'mean_abs_correlation': round(float(np.nanmean(np.abs(corrs))), 3),
         'fps': fps, 'sr': sr}
    mgf = MgFigure(figure=fig, figure_type='video.body_audio_coupling', data=d, layers=None, image=target_name)
    self.body_audio_coupling_figure = mgf
    return mgf

dynamics_coupling

dynamics_coupling(fs=50.0, max_lag=2.0, dpi=300, autoshow=True, title=None, target_name=None, overwrite=True)

Compare audio loudness with movement quantity — does the dancer move more when the music is louder?

Aligns the audio RMS-loudness envelope with the quantity-of-motion envelope and reports their correlation (at zero lag and at the best lag within max_lag seconds). The figure overlays the two normalised envelopes and shows a scatter of loudness vs. motion.

Returns an MgFigure (metrics in .data), or None if the video has no audio.

Source code in musicalgestures/_audio_video.py

def mg_dynamics_coupling(self, fs: float = 50.0, max_lag: float = 2.0, dpi: int = 300, autoshow: bool = True,
                         title: str | None = None, target_name: str | None = None, overwrite: bool = True) -> "MgFigure":
    """
    Compare audio **loudness** with movement **quantity** — does the dancer move more when the
    music is louder?

    Aligns the audio RMS-loudness envelope with the quantity-of-motion envelope and reports their
    correlation (at zero lag and at the best lag within ``max_lag`` seconds). The figure overlays
    the two normalised envelopes and shows a scatter of loudness vs. motion.

    Returns an MgFigure (metrics in ``.data``), or None if the video has no audio.
    """
    from musicalgestures._movementbeats import _movement_qom

    if not has_audio(self.filename):
        print('The video has no audio track.')
        return None

    target_name = resolve_filename(self.of, '_dynamics_coupling.png', target_name, overwrite)

    rms, t_a, sr = _audio_env(self, 'rms')
    qom, fps = _movement_qom(self)
    t_m = np.arange(len(qom)) / max(fps, 1e-9)
    t, a, m = _common_grid(t_a, rms, t_m, qom, fs)
    if t is None or len(t) < 8:
        print('Audio and movement do not overlap enough in time.')
        return None
    az, mz = _z(a), _z(m)

    zero_r = _safe_corr(az, mz)
    # Best lag within +/- max_lag seconds
    max_shift = int(max_lag * fs)
    best_r, best_lag = zero_r, 0.0
    for s in range(-max_shift, max_shift + 1):
        if s < 0:
            r = _safe_corr(az[-s:], mz[:len(mz) + s])
        elif s > 0:
            r = _safe_corr(az[:len(az) - s], mz[s:])
        else:
            r = zero_r
        if r > best_r:
            best_r, best_lag = r, s / fs

    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5), dpi=dpi,
                                   gridspec_kw={'width_ratios': [2, 1]})
    fig.patch.set_facecolor('white')
    if title == 'filename':
        title = os.path.basename(self.filename)
    fig.suptitle(title or 'Audio loudness vs. movement quantity', fontsize=14)

    ax1.plot(t, az, color='#1f77b4', lw=0.9, label='Audio loudness (RMS)')
    ax1.plot(t, mz, color='#d62728', lw=0.9, alpha=0.8, label='Quantity of motion')
    ax1.set_xlabel('Time (s)'); ax1.set_ylabel('Normalised')
    ax1.legend(loc='upper right', fontsize=8)
    ax1.set_title(f'Zero-lag r = {zero_r:.2f}   |   best r = {best_r:.2f} @ {best_lag:+.2f}s',
                  fontsize=10)

    ax2.scatter(az, mz, s=4, alpha=0.3, color='#444444')
    ax2.set_xlabel('Audio loudness (z)'); ax2.set_ylabel('Quantity of motion (z)')
    ax2.set_title('Loudness vs. motion', fontsize=10)

    plt.tight_layout(rect=[0, 0, 1, 0.95])
    plt.savefig(target_name, format='png', transparent=False)
    plt.close(fig)

    d = {'zero_lag_corr': round(zero_r, 3), 'best_corr': round(best_r, 3),
         'best_lag_s': round(best_lag, 3), 'fps': fps, 'sr': sr}
    mgf = MgFigure(figure=fig, figure_type='video.dynamics_coupling', data=d, layers=None, image=target_name)
    self.dynamics_coupling_figure = mgf
    return mgf

pose

pose(model='mediapipe', device='gpu', threshold=0.1, downsampling_factor=2, use_cache=True, save_data=True, data_format='csv', save_video=True, style='both', overlay=True, background='black', convert=None, quiet=True, marker_history=0, save_average_pose=True, save_trajectories=True, transparent_trajectories=None, trajectory_background=None, trajectory_labels=False, target_name_video=None, target_name_data=None, target_name_average=None, target_name_trajectories=None, overwrite=True)

Renders a video with the pose estimation (aka. "keypoint detection" or "skeleton tracking") overlaid on it. Outputs the predictions in a text file containing the normalized x and y coordinates of each keypoint (default format is csv).

Supports two backends:

• MediaPipe (model='mediapipe'): Uses Google's MediaPipe Pose which detects 33 landmarks. Runs on CPU, or on GPU via MediaPipe's GPU delegate when device='gpu' (with automatic CPU fallback if the delegate is unavailable). Requires the optional mediapipe package (pip install musicalgestures[pose]). On first use, the model file (~8–28 MB) is downloaded automatically and cached in musicalgestures/models/.
• OpenPose (model='body_25', 'coco', or 'mpi'): Uses Caffe-based OpenPose models. Model weights (~200 MB) are downloaded on first use. GPU here requires an OpenCV built with CUDA; if unavailable while device='gpu', pose() automatically switches to the MediaPipe backend (when installed) for GPU acceleration.

Parameters:

Name	Type	Description	Default
model	str	Pose model to use. 'mediapipe' (default) uses MediaPipe Pose (33 landmarks with depth + visibility, model auto-downloaded on first use); it is fast on plain CPU, needs no CUDA build, and is best for single-person analysis. 'body_25' loads the OpenPose BODY_25 model (25 keypoints), 'mpi' loads the MPII model (15 keypoints), 'coco' loads the COCO model (18 keypoints). The OpenPose models support multi-person scenes but are slow without a CUDA-enabled OpenCV build. Defaults to 'mediapipe'.	'mediapipe'
device	str	Compute backend ('cpu' or 'gpu'). For OpenPose models this selects the OpenCV DNN backend (GPU needs a CUDA-enabled OpenCV). For MediaPipe it selects the inference delegate (GPU delegate with CPU fallback). Defaults to 'gpu'.	'gpu'
threshold	float	The normalized confidence threshold that decides whether we keep or discard a predicted point. Discarded points get substituted with (0, 0) in the output data. Defaults to 0.1.	0.1
downsampling_factor	int	Decides how much we downsample the video before we pass it to the neural network. Ignored when model='mediapipe'. Defaults to 2.	2
use_cache	bool	If True (default), reuse keypoints from a previous pose() run on this object (same model/threshold) to re-render a different style/overlay/background without re-running the network — e.g. run style='markers' then style='skeleton' fast. Defaults to True.	True
save_data	bool	Whether we save the predicted pose data to a file. Defaults to True.	True
data_format	str	Specifies format of pose-data. Accepted values are 'csv', 'tsv', 'txt' and 'c3d' (motion-capture format; requires the optional c3d package). For multiple output formats, use a list, e.g. ['csv', 'c3d']. Defaults to 'csv'.	'csv'
save_video	bool	Whether we save the video with the estimated pose overlaid on it. Defaults to True.	True
style	str	How to draw the pose. 'both' draws markers (keypoints) connected by joint lines (the skeleton); 'markers' draws only the keypoints; 'skeleton' draws only the connecting joint lines. Defaults to 'both'.	'both'
overlay	bool	If True, draw the pose on top of the original video frames. If False, draw it on a plain background instead (a "markers only" video with no video underneath). Defaults to True.	True
background	str	Background colour used when overlay=False: 'black' (default) or 'white'. With 'white' the skeleton and markers are drawn in black (an inverted, print-friendly look). With 'black' they are drawn in bright colours. Ignored when overlay=True.	'black'
marker_history	int	If greater than 0, draw a motion trail for every marker by joining its positions over the last marker_history frames. Defaults to 0 (no trails). Works in all rendering paths (OpenPose, MediaPipe, and cached re-render).	0
convert	bool	Whether non-AVI input is first converted to an all-intra MJPEG .avi (cached as self.as_avi) for frame-accurate decoding. Defaults to None ("auto"): the MediaPipe backend reads the source file directly (it decodes sequentially through an FFmpeg pipe and needs no intra-frame AVI), while the OpenPose backend converts. Pass True/False to force the behaviour.	None
quiet	bool	MediaPipe only. If True (default), suppress MediaPipe's native C++/GL console logs (EGL init, absl INFO/WARNING, GPU-delegate messages) during inference. Set to False to see them for debugging.	True
target_name_video	str	Target output name for the video. Defaults to None (which assumes that the input filename with the suffix "_pose" should be used).	None
save_average_pose	bool	Whether to also render an image of the average pose over the whole video, with each marker coloured/labelled by its average quantity of motion (px/frame) and labelled with its dominant movement frequency (Hz). A CSV of the per-marker statistics is saved alongside it. Defaults to True.	True
save_trajectories	bool	Whether to also render an image of every marker's spatial trajectory across the whole video. Defaults to True.	True
trajectory_labels	bool	Whether to annotate the trajectories image with each marker's name. Defaults to False (cleaner image).	False
trajectory_background	str	Background of the trajectories PNG: 'black', 'white', or 'transparent' (for overlaying on the video). Defaults to None ("auto"): transparent when the trajectories image is the only one exported, else black. Takes precedence over the legacy transparent_trajectories flag.	None
target_name_data	str	Target output name for the data. Defaults to None (which assumes that the input filename with the suffix "_pose" should be used).	None
target_name_average	str	Target output name for the average-pose image. Defaults to None (input filename with the suffix "_pose_average.png").	None
target_name_trajectories	str	Target output name for the trajectories image. Defaults to None (input filename with the suffix "_pose_trajectories.png").	None
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filenames to avoid overwriting. Defaults to True.	True

Returns:

Name	Type	Description
MgVideo	'musicalgestures.MgVideo'	An MgVideo pointing to the output video. The average-pose and trajectories images (when rendered) are attached as .average_pose and .trajectories (MgImage), and the collected keypoints are available on the parent object as self.pose_average / self.pose_trajectories.

Source code in musicalgestures/_pose.py

def pose(
        self,
        model: str = 'mediapipe',
        device: str = 'gpu',
        threshold: float = 0.1,
        downsampling_factor: int = 2,
        use_cache: bool = True,
        save_data: bool = True,
        data_format: str | list = 'csv',
        save_video: bool = True,
        style: str = 'both',
        overlay: bool = True,
        background: str = 'black',
        convert: bool | None = None,
        quiet: bool = True,
        marker_history: int = 0,
        save_average_pose: bool = True,
        save_trajectories: bool = True,
        transparent_trajectories: bool | None = None,
        trajectory_background: str | None = None,
        trajectory_labels: bool = False,
        target_name_video: str | None = None,
        target_name_data: str | None = None,
        target_name_average: str | None = None,
        target_name_trajectories: str | None = None,
        overwrite: bool = True) -> "musicalgestures.MgVideo":
    """
    Renders a video with the pose estimation (aka. "keypoint detection" or "skeleton tracking") overlaid on it.
    Outputs the predictions in a text file containing the normalized x and y coordinates of each keypoint
    (default format is csv).

    Supports two backends:

    * **MediaPipe** (``model='mediapipe'``): Uses Google's MediaPipe Pose which detects 33
      landmarks. Runs on CPU, or on GPU via MediaPipe's GPU delegate when ``device='gpu'``
      (with automatic CPU fallback if the delegate is unavailable). Requires the optional
      ``mediapipe`` package (``pip install musicalgestures[pose]``). On first use, the model
      file (~8–28 MB) is downloaded automatically and cached in ``musicalgestures/models/``.
    * **OpenPose** (``model='body_25'``, ``'coco'``, or ``'mpi'``): Uses Caffe-based OpenPose
      models.  Model weights (~200 MB) are downloaded on first use. GPU here requires an
      OpenCV built with CUDA; if unavailable while ``device='gpu'``, ``pose()`` automatically
      switches to the MediaPipe backend (when installed) for GPU acceleration.

    Args:
        model (str, optional): Pose model to use. ``'mediapipe'`` (default) uses MediaPipe Pose (33
            landmarks with depth + visibility, model auto-downloaded on first use); it is fast on plain
            CPU, needs no CUDA build, and is best for single-person analysis. ``'body_25'`` loads the
            OpenPose BODY_25 model (25 keypoints), ``'mpi'`` loads the MPII model (15 keypoints),
            ``'coco'`` loads the COCO model (18 keypoints). The OpenPose models support multi-person
            scenes but are slow without a CUDA-enabled OpenCV build. Defaults to 'mediapipe'.
        device (str, optional): Compute backend ('cpu' or 'gpu'). For OpenPose models this
            selects the OpenCV DNN backend (GPU needs a CUDA-enabled OpenCV). For MediaPipe
            it selects the inference delegate (GPU delegate with CPU fallback). Defaults to 'gpu'.
        threshold (float, optional): The normalized confidence threshold that decides whether we
            keep or discard a predicted point. Discarded points get substituted with (0, 0) in the
            output data. Defaults to 0.1.
        downsampling_factor (int, optional): Decides how much we downsample the video before we
            pass it to the neural network. Ignored when ``model='mediapipe'``. Defaults to 2.
        use_cache (bool, optional): If True (default), reuse keypoints from a previous pose() run on
            this object (same model/threshold) to re-render a different `style`/`overlay`/`background`
            without re-running the network — e.g. run `style='markers'` then `style='skeleton'` fast.
            Defaults to True.
        save_data (bool, optional): Whether we save the predicted pose data to a file. Defaults to True.
        data_format (str, optional): Specifies format of pose-data. Accepted values are 'csv', 'tsv',
            'txt' and 'c3d' (motion-capture format; requires the optional ``c3d`` package). For multiple
            output formats, use a list, e.g. ['csv', 'c3d']. Defaults to 'csv'.
        save_video (bool, optional): Whether we save the video with the estimated pose overlaid on it.
            Defaults to True.
        style (str, optional): How to draw the pose. `'both'` draws markers (keypoints) connected by
            joint lines (the skeleton); `'markers'` draws only the keypoints; `'skeleton'` draws only
            the connecting joint lines. Defaults to 'both'.
        overlay (bool, optional): If True, draw the pose on top of the original video frames. If False,
            draw it on a plain background instead (a "markers only" video with no video underneath).
            Defaults to True.
        background (str, optional): Background colour used when `overlay=False`: `'black'` (default) or
            `'white'`. With `'white'` the skeleton and markers are drawn in black (an inverted, print-friendly
            look). With `'black'` they are drawn in bright colours. Ignored when `overlay=True`.
        marker_history (int, optional): If greater than 0, draw a motion trail for every marker by joining its
            positions over the last `marker_history` frames. Defaults to 0 (no trails). Works in all rendering
            paths (OpenPose, MediaPipe, and cached re-render).
        convert (bool, optional): Whether non-AVI input is first converted to an all-intra MJPEG `.avi`
            (cached as ``self.as_avi``) for frame-accurate decoding. Defaults to None ("auto"): the
            MediaPipe backend reads the source file directly (it decodes sequentially through an FFmpeg
            pipe and needs no intra-frame AVI), while the OpenPose backend converts. Pass True/False to
            force the behaviour.
        quiet (bool, optional): MediaPipe only. If True (default), suppress MediaPipe's native C++/GL
            console logs (EGL init, absl INFO/WARNING, GPU-delegate messages) during inference. Set to
            False to see them for debugging.
        target_name_video (str, optional): Target output name for the video. Defaults to None (which
            assumes that the input filename with the suffix "_pose" should be used).
        save_average_pose (bool, optional): Whether to also render an image of the average pose over
            the whole video, with each marker coloured/labelled by its average quantity of motion
            (px/frame) and labelled with its dominant movement frequency (Hz). A CSV of the per-marker
            statistics is saved alongside it. Defaults to True.
        save_trajectories (bool, optional): Whether to also render an image of every marker's spatial
            trajectory across the whole video. Defaults to True.
        trajectory_labels (bool, optional): Whether to annotate the trajectories image with each
            marker's name. Defaults to False (cleaner image).
        trajectory_background (str, optional): Background of the trajectories PNG: ``'black'``,
            ``'white'``, or ``'transparent'`` (for overlaying on the video). Defaults to None
            ("auto"): transparent when the trajectories image is the only one exported, else black.
            Takes precedence over the legacy ``transparent_trajectories`` flag.
        target_name_data (str, optional): Target output name for the data. Defaults to None (which
            assumes that the input filename with the suffix "_pose" should be used).
        target_name_average (str, optional): Target output name for the average-pose image. Defaults
            to None (input filename with the suffix "_pose_average.png").
        target_name_trajectories (str, optional): Target output name for the trajectories image.
            Defaults to None (input filename with the suffix "_pose_trajectories.png").
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically
            increment target filenames to avoid overwriting. Defaults to True.

    Returns:
        MgVideo: An MgVideo pointing to the output video. The average-pose and trajectories images
            (when rendered) are attached as ``.average_pose`` and ``.trajectories`` (MgImage), and the
            collected keypoints are available on the parent object as ``self.pose_average`` /
            ``self.pose_trajectories``.
    """

    style = str(style).lower()
    if style not in ('both', 'markers', 'skeleton'):
        print(f"Unrecognized style '{style}', falling back to 'both'. Use 'both', 'markers' or 'skeleton'.")
        style = 'both'

    background = str(background).lower()
    if background not in ('black', 'white'):
        print(f"Unrecognized background '{background}', falling back to 'black'. Use 'black' or 'white'.")
        background = 'black'

    # --- MediaPipe backend ---------------------------------------------------
    # Explicit MediaPipe request, or auto-preference: when GPU is requested for an
    # OpenPose model but OpenCV has no CUDA backend, fall back to the (fast, reliable)
    # MediaPipe pose backend instead of CPU OpenPose. We use the CPU delegate here:
    # MediaPipe's GPU delegate is the OpenGL-ES path (the integrated GPU on Linux, not
    # an NVIDIA card) and is fragile, so it is reserved for explicit model='mediapipe',
    # device='gpu' requests.
    use_mediapipe = model.lower() == 'mediapipe'
    mediapipe_device = device
    if not use_mediapipe and device.lower() == 'gpu' and not in_colab() and get_cuda_device_count() <= 0:
        if _mediapipe_available():
            print(
                f"GPU requested but OpenCV has no CUDA backend; switching from '{model}' to the "
                "MediaPipe pose backend (CPU/XNNPACK — fast and reliable).\n  "
                + cuda_unavailable_reason()
            )
            use_mediapipe = True
            mediapipe_device = 'cpu'
        # else: fall through to the OpenPose path, which will warn and use CPU.

    # MediaPipe is an optional dependency (the `[pose]` extra). If it was selected (it is the
    # default backend) but isn't installed, fall back to the OpenPose BODY_25 model — it runs on
    # the always-present OpenCV DNN and auto-downloads its weights — so pose() works on a bare
    # `pip install musicalgestures`.
    #
    # That fallback assumed OpenCV could always load a Caffe model, which
    # stopped being true in OpenCV 5.0. Where it is not true there is nowhere
    # to fall back *to*, and announcing a fallback that then fails on the
    # backend it fell back to is worse than saying so plainly.
    if use_mediapipe and not _mediapipe_available():
        if not caffe_supported():
            raise MgDependencyError(
                f"pose() needs MediaPipe on this machine. The default backend "
                f"is not installed, and the OpenPose fallback cannot run "
                f"either: its models are Caffe, and OpenCV {cv2.__version__} "
                f"dropped the Caffe importer in 5.0. Install it with "
                f"`pip install musicalgestures[pose]`, or install OpenCV 4 "
                f"(`pip install 'opencv-python<5'`) to use the OpenPose "
                f"skeletons instead."
            )
        print("MediaPipe is not installed; falling back to the OpenPose 'body_25' backend. "
              "Install MediaPipe for the default backend with: pip install musicalgestures[pose]")
        use_mediapipe = False
        if model.lower() == 'mediapipe':
            model = 'body_25'

    # Resolve the "auto" convert default: MediaPipe reads the source directly through an
    # FFmpeg pipe and needs no all-intra AVI; OpenPose keeps the frame-accurate conversion.
    if convert is None:
        convert = not use_mediapipe

    # --- Reuse cached keypoints (skip re-inference) --------------------------
    # If a previous pose() run on this object used the same model/threshold, re-render
    # from the stored keypoints with the new style/overlay/background instead of running
    # the (expensive) network again.
    effective_model = 'mediapipe' if use_mediapipe else model.lower()
    cache = getattr(self, '_pose_keypoints', None)
    if use_cache and cache is not None \
            and cache.get('model') == effective_model \
            and cache.get('threshold') == threshold \
            and cache.get('downsampling_factor') == (None if use_mediapipe else downsampling_factor):
        return _rerender_pose_from_cache(
            self, style=style, overlay=overlay, background=background,
            save_data=save_data, data_format=data_format, save_video=save_video,
            save_average_pose=save_average_pose, save_trajectories=save_trajectories,
            transparent_trajectories=transparent_trajectories,
            trajectory_background=trajectory_background, trajectory_labels=trajectory_labels,
            marker_history=marker_history,
            target_name_video=target_name_video, target_name_data=target_name_data,
            target_name_average=target_name_average, target_name_trajectories=target_name_trajectories,
            overwrite=overwrite)

    if use_mediapipe:
        return _pose_mediapipe(
            self,
            device=mediapipe_device,
            threshold=threshold,
            save_data=save_data,
            data_format=data_format,
            save_video=save_video,
            style=style,
            overlay=overlay,
            background=background,
            convert=convert,
            quiet=quiet,
            marker_history=marker_history,
            save_average_pose=save_average_pose,
            save_trajectories=save_trajectories,
            transparent_trajectories=transparent_trajectories,
            trajectory_background=trajectory_background,
            trajectory_labels=trajectory_labels,
            target_name_video=target_name_video,
            target_name_data=target_name_data,
            target_name_average=target_name_average,
            target_name_trajectories=target_name_trajectories,
            overwrite=overwrite,
        )
    # -------------------------------------------------------------------------

    # The OpenPose backends are Caffe models, and OpenCV removed its Caffe
    # importer in 5.0 -- `readNetFromCaffe` is gone and `readNet` refuses the
    # format outright. Checked here, before the weights are looked for, so an
    # incompatible environment is not discovered after a 200 MB download and
    # a full decode; the failure used to surface as `AttributeError: module
    # 'cv2.dnn' has no attribute 'readNetFromCaffe'` from deep inside the run.
    #
    # Not silently switched to MediaPipe. Its 33 landmarks are a different
    # skeleton from BODY_25's, COCO's or MPI's, so a substituted backend would
    # return data that looks like what was asked for and is not.
    _require_caffe_support()

    module_path = os.path.abspath(os.path.dirname(musicalgestures.__file__))

    if model.lower() == 'mpi':
        protoFile = module_path + '/pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt'
        weightsFile = module_path + '/pose/mpi/pose_iter_160000.caffemodel'
        model = 'mpi'
        nPoints = 15
        POSE_PAIRS = [[0, 1], [1, 2], [2, 3], [3, 4], [1, 5], [5, 6], [6, 7], [
            1, 14], [14, 8], [8, 9], [9, 10], [14, 11], [11, 12], [12, 13]]
    elif model.lower() == 'coco':
        protoFile = module_path + '/pose/coco/pose_deploy_linevec.prototxt'
        weightsFile = module_path + '/pose/coco/pose_iter_440000.caffemodel'
        model = 'coco'
        nPoints = 18
        POSE_PAIRS = [[1, 0], [1, 2], [1, 5], [2, 3], [3, 4], [5, 6], [6, 7], [1, 8], [
            8, 9], [9, 10], [1, 11], [11, 12], [12, 13], [0, 14], [0, 15], [14, 16], [15, 17]]
    elif model.lower() == 'body_25':
        protoFile = module_path + '/pose/body_25/pose_deploy.prototxt'
        weightsFile = module_path + '/pose/body_25/pose_iter_584000.caffemodel'
        model = 'body_25'
        nPoints = 25
        POSE_PAIRS = [[1, 8], [1, 2], [1, 5], [2, 3], [3, 4], [5, 6], [6, 7], [8, 9], [9, 10], [10, 11], [8, 12], [12, 13], [
            13, 14], [1, 0], [0, 15], [15, 17], [0, 16], [16, 18], [14, 19], [19, 20], [14, 21], [11, 22], [22, 23], [11, 24]]
    else:
        print(f'Unrecognized model "{model}", switching to default (mpi).')
        protoFile = module_path + '/pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt'
        weightsFile = module_path + '/pose/mpi/pose_iter_160000.caffemodel'
        model = 'mpi'

    # Check if .caffemodel file exists, download if necessary
    if not os.path.exists(weightsFile):
        print('Could not find weights file.')
        # Notebook/nbclient runs cannot satisfy input(), so auto-download in non-interactive mode.
        if not sys.stdin or not sys.stdin.isatty():
            print('Non-interactive session detected. Downloading model weights automatically (~200MB).')
            download_model(model)
        else:
            print('Do you want to download it (~200MB)? (y/n)')
            answer = input()
            if answer.lower() == 'n':
                print('Ok. Exiting...')
                return musicalgestures.MgVideo(self.filename, color=self.color, returned_by_process=True)
            elif answer.lower() == 'y':
                download_model(model)
            else:
                print(f'Unrecognized answer "{answer}". Exiting...')
                return musicalgestures.MgVideo(self.filename, color=self.color, returned_by_process=True)

        if not os.path.exists(weightsFile):
            print('Model weights are still missing after download attempt. Exiting pose() call.')
            return musicalgestures.MgVideo(self.filename, color=self.color, returned_by_process=True)

    # Read the network into Memory
    net = cv2.dnn.readNetFromCaffe(protoFile, weightsFile)
    device = device.lower()
    # enforce CPU device in Colab
    if in_colab() and device == 'gpu':
        print('Sorry, OpenCV GPU acceleration is not supported in Colab. Switching to CPU.')
        device = 'cpu'
    elif device == 'gpu':
        if get_cuda_device_count() <= 0:
            print('OpenCV CUDA backend is unavailable. Switching to CPU.\n  ' + cuda_unavailable_reason())
            device = 'cpu'

    if device == "cpu":
        net.setPreferableBackend(cv2.dnn.DNN_TARGET_CPU)
    elif device == "gpu":
        net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
        net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)
    else:
        print(f'Unrecognized device "{device}", switching to default (cpu).')
        net.setPreferableBackend(cv2.dnn.DNN_TARGET_CPU)

    of, fex = os.path.splitext(self.filename)
    # Write the result in the original container so we don't produce an .avi that then has
    # to be converted to .mp4; the AVI is only an intermediate for frame-accurate decoding.
    output_fex = fex

    if convert and fex.lower() != '.avi':
        # first check if there already is a converted version, if not create one and register it to the parent self
        if "as_avi" not in self.__dict__.keys():
            file_as_avi = convert_to_avi(of + fex, overwrite=overwrite)
            # register it as the avi version for the file
            self.as_avi = musicalgestures.MgVideo(file_as_avi)
        # point of and fex to the avi version
        of, fex = self.as_avi.of, self.as_avi.fex
        filename = of + fex
    else:
        # use the source file directly (e.g. an mp4 that decodes frame-accurately)
        filename = self.filename

    inWidth = int(roundup(self.width/downsampling_factor, 2))
    inHeight = int(roundup(self.height/downsampling_factor, 2))

    pb = MgProgressbar(total=self.length, prefix='Rendering pose estimation video:')

    if save_video:
        if target_name_video is None:
            target_name_video = of + '_pose' + output_fex
        else:
            target_name_video = os.path.splitext(target_name_video)[0] + output_fex
        if not overwrite:
            target_name_video = generate_outfilename(target_name_video)

    # Pipe video with FFmpeg for reading frame by frame
    cmd = ['ffmpeg', '-y', '-i', filename] # define ffmpeg command        
    process = ffmpeg_cmd(cmd, total_time=self.length, pipe='read')
    video_out = None

    ii = 0
    data = []
    # Accumulate the average frame as a background for the average-pose image
    collect_extra = save_average_pose or save_trajectories
    avg_acc = np.zeros((self.height, self.width, 3), dtype=np.float64) if save_average_pose else None
    avg_n = 0
    from collections import deque
    _trail = deque(maxlen=int(marker_history)) if marker_history and marker_history > 0 else None

    while True:
        # Read frame-by-frame
        out = process.stdout.read(self.width*self.height*3)

        if out == b'':
            pb.progress(self.length)
            break

        # Transform the bytes read into a numpy array
        frame = np.frombuffer(out, dtype=np.uint8).reshape([self.height, self.width, 3]).copy() # height, width, channels

        if avg_acc is not None:
            avg_acc += frame
            avg_n += 1

        inpBlob = cv2.dnn.blobFromImage(frame, 1.0 / 255, (inWidth, inHeight), (0, 0, 0), swapRB=False, crop=False)
        net.setInput(inpBlob)
        output = net.forward()

        H = output.shape[2]
        W = output.shape[3]
        points = []

        for i in range(nPoints):

            # confidence map of corresponding body's part.
            probMap = output[0, i, :, :]

            # Find global maxima of the probMap.
            minVal, prob, minLoc, point = cv2.minMaxLoc(probMap)

            # Scale the point to fit on the original image
            x = (self.width * point[0]) / W
            y = (self.height * point[1]) / H

            if prob > threshold:
                points.append((int(x), int(y)))

            else:
                points.append(None)

        # Always collect keypoints so the average-pose/trajectories images and the
        # keypoint cache (for fast re-rendering) are available; file-writing is gated below.
        time = frame2ms(ii, self.fps)
        points_list = [[list(point)[0]/self.width, list(point)[1]/self.height, ] if point is not None else [
            0, 0] for point in points]
        points_list_flat = itertools.chain.from_iterable(points_list)
        datapoint = [time]
        datapoint += points_list_flat
        data.append(datapoint)

        # Draw on the video frame, or on a plain canvas when overlay is disabled
        canvas, line_color, marker_color = _pose_canvas_and_colors(frame, overlay, background)

        # Marker history trails (last N frames)
        if _trail is not None:
            _trail.append([[p[0], p[1]] if p is not None else [np.nan, np.nan] for p in points])
            _draw_marker_trails(canvas, _trail, marker_color)

        # Joint lines (skeleton)
        if style in ('both', 'skeleton'):
            for pair in POSE_PAIRS:
                partA, partB = pair[0], pair[1]
                if points[partA] and points[partB]:
                    cv2.line(canvas, points[partA], points[partB],
                             line_color, 2, lineType=cv2.LINE_AA)

        # Markers (keypoints)
        if style in ('both', 'markers'):
            for point in points:
                if point is not None:
                    cv2.circle(canvas, point, 4, marker_color, thickness=-1, lineType=cv2.FILLED)

        frame = canvas

        if save_video:
            if video_out is None:
                cmd =['ffmpeg', '-y', '-s', '{}x{}'.format(frame.shape[1], frame.shape[0]), 
                    '-r', str(self.fps), '-f', 'rawvideo', '-pix_fmt', 'bgr24', '-vcodec', 'rawvideo', 
                    '-i', '-', '-vcodec', 'libx264', '-pix_fmt', 'yuv420p', target_name_video]
                video_out = ffmpeg_cmd(cmd, total_time=self.length, pipe='write')

            video_out.stdin.write(frame.astype(np.uint8))

        # Flush the buffer
        process.stdout.flush()
        pb.progress(ii)
        ii += 1

    # Terminate the processes
    if save_video:
        video_out.stdin.close()
        video_out.wait()
        # Check if the original video fil has audio
        if self.has_audio:
            source_audio = extract_wav(of + fex)
            embed_audio_in_video(source_audio, target_name_video)
            os.remove(source_audio)

    process.terminate()

    def save_txt(of, width, height, model, data, data_format, target_name_data, overwrite):
        """
        Helper function to export pose estimation data as textfile(s).
        """
        def save_single_file(of, width, height, model, data, data_format, target_name_data, overwrite):
            """
            Helper function to export pose estimation data as a textfile using pandas.
            """

            coco_table = ['Nose', 'Neck', 'Right Shoulder', 'Right Elbow', 'Right Wrist', 'Left Shoulder', 'Left Elbow', 'Left Wrist', 'Right Hip',
                          'Right Knee', 'Right Ankle', 'Left Hip', 'Left Knee', 'Left Ankle', 'Right Eye', 'Left Eye', 'Right Ear', 'Left Ear']
            mpi_table = ['Head', 'Neck', 'Right Shoulder', 'Right Elbow', 'Right Wrist', 'Left Shoulder', 'Left Elbow',
                         'Left Wrist', 'Right Hip', 'Right Knee', 'Right Ankle', 'Left Hip', 'Left Knee', 'Left Ankle', 'Chest']
            body_25_table = ['Nose', 'Neck', 'Right Shoulder', 'Right Elbow', 'Right Wrist', 'Left Shoulder', 'Left Elbow', 'Left Wrist', 'Mid Hip', 'Right Hip', 'Right Knee', 'Right Ankle', 'Left Hip',
                             'Left Knee', 'Left Ankle', 'Right Eye', 'Left Eye', 'Right Ear', 'Left Ear', "Left Big Toe", "Left Small Toe", "Left Heel", "Right Big Toe", "Right Small Toe", "Right Heel"]
            headers = ['Time']

            table_to_use = []
            if model.lower() == 'mpi':
                table_to_use = mpi_table
            elif model.lower() == 'coco':
                table_to_use = coco_table
            elif model.lower() == 'body_25':
                table_to_use = body_25_table

            for i in range(len(table_to_use)):
                header_x = table_to_use[i] + ' X'
                header_y = table_to_use[i] + ' Y'
                headers.append(header_x)
                headers.append(header_y)

            data_format = data_format.lower()

            df = pd.DataFrame(data=data, columns=headers)

            if data_format == "tsv":

                if target_name_data is None:
                    target_name_data = of+'_pose.tsv'
                else:
                    # take name, but enforce tsv
                    target_name_data = os.path.splitext(
                        target_name_data)[0] + '.tsv'
                if not overwrite:
                    target_name_data = generate_outfilename(target_name_data)

                with open(target_name_data, 'wb') as f:
                    head_str = ''
                    for head in headers:
                        head_str += head + '\t'
                    head_str += '\n'
                    f.write(head_str.encode())
                    fmt_list = ['%d']
                    fmt_list += ['%.15f' for item in range(
                        len(table_to_use)*2)]
                    np.savetxt(f, df.values, delimiter='\t', fmt=fmt_list)

            elif data_format == "csv":

                if target_name_data is None:
                    target_name_data = of+'_pose.csv'
                else:
                    # take name, but enforce csv
                    target_name_data = os.path.splitext(
                        target_name_data)[0] + '.csv'
                if not overwrite:
                    target_name_data = generate_outfilename(target_name_data)

                df.to_csv(target_name_data, index=None)

            elif data_format == "txt":

                if target_name_data is None:
                    target_name_data = of+'_pose.txt'
                else:
                    # take name, but enforce txt
                    target_name_data = os.path.splitext(
                        target_name_data)[0] + '.txt'
                if not overwrite:
                    target_name_data = generate_outfilename(target_name_data)

                with open(target_name_data, 'wb') as f:
                    head_str = ''
                    for head in headers:
                        head_str += head + ' '
                    head_str += '\n'
                    f.write(head_str.encode())
                    fmt_list = ['%d']
                    fmt_list += ['%.15f' for item in range(
                        len(table_to_use)*2)]
                    np.savetxt(f, df.values, delimiter=' ', fmt=fmt_list)
            elif data_format not in ["tsv", "csv", "txt"]:
                print(
                    f"Invalid data format: '{data_format}'.\nFalling back to '.csv'.")
                save_single_file(of, width, height, model, data, "csv",
                                 target_name_data=target_name_data, overwrite=overwrite)

        if type(data_format) == str:
            save_single_file(of, width, height, model, data, data_format,
                             target_name_data=target_name_data, overwrite=overwrite)

        elif type(data_format) == list:
            if all([item.lower() in ["csv", "tsv", "txt"] for item in data_format]):
                data_format = list(set(data_format))
                [save_single_file(of, width, height, model, data, item, target_name_data=target_name_data, overwrite=overwrite)
                 for item in data_format]
            else:
                print(
                    f"Unsupported formats in {data_format}.\nFalling back to '.csv'.")
                save_single_file(of, width, height, model, data, "csv",
                                 target_name_data=target_name_data, overwrite=overwrite)

    if save_data:
        text_format = _handle_c3d(of, data, OPENPOSE_NAMES.get(model.lower()), self.fps,
                                  self.width, self.height, data_format, target_name_data, overwrite)
        if text_format is not None:
            save_txt(of, self.width, self.height, model, data, text_format,
                     target_name_data=target_name_data, overwrite=overwrite)

    # Render the average-pose and trajectories images from the collected keypoints
    names = OPENPOSE_NAMES.get(model.lower())
    # Cache keypoints so a later pose() call can re-render a different style without re-inference
    self._pose_keypoints = {
        'model': model.lower(), 'threshold': threshold, 'downsampling_factor': downsampling_factor,
        'names': names, 'connections': POSE_PAIRS, 'data': data,
        'width': self.width, 'height': self.height, 'fps': self.fps,
        'filename': filename, 'of': of, 'fex': fex, 'output_fex': output_fex,
        'has_audio': self.has_audio,
    }
    avg_frame = (avg_acc / avg_n).astype(np.uint8) if (avg_acc is not None and avg_n > 0) else None
    average_image, trajectories_image = _render_pose_extras(
        data, names, POSE_PAIRS, self.width, self.height, self.fps,
        avg_frame, of, save_average_pose, save_trajectories,
        target_name_average, target_name_trajectories, overwrite,
        transparent_trajectories=transparent_trajectories,
        trajectory_background=trajectory_background,
        trajectory_labels=trajectory_labels, style=style, background=background)
    self.pose_average = average_image
    self.pose_trajectories = trajectories_image

    if save_video:
        # save result as pose_video for parent MgVideo
        self.pose_video = musicalgestures.MgVideo(target_name_video, color=self.color, returned_by_process=True)
        self.pose_video.average_pose = average_image
        self.pose_video.trajectories = trajectories_image
        return self.pose_video
    else:
        # otherwise just return the parent MgVideo
        return self

pose_waterfall

pose_waterfall(style='trajectories', n_samples=40, markers=None, color_by=None, cmap='hsv', dpi=200, elev=20, azim=-60, lw=1.0, axes=True, crop=False, target_name=None, overwrite=True, **pose_kwargs)

Render a 3D spatio-temporal waterfall of the pose, cascading along the time (depth) axis — a pose-based counterpart to silhouette_waterfall(). Uses cached pose keypoints from a previous pose() call when available; otherwise it runs pose estimation first (extra keyword arguments such as model/device/downsampling_factor are forwarded to pose()).

Parameters:

Name	Type	Description	Default
style	str	What to draw. 'trajectories' (default) draws each marker's continuous path through (x, time, y); 'markers' scatters the markers at n_samples time slices; 'skeleton' draws the skeleton joint lines at each time slice; 'both' draws markers + skeleton.	'trajectories'
n_samples	int	Number of time slices for the marker/skeleton styles. Defaults to 40 (ignored for 'trajectories').	40
markers	list	Subset of marker names or indices to draw. Defaults to all.	None
color_by	str	'marker' or 'time'. Defaults to None ("auto"): 'marker' for trajectories, 'time' for the slice styles.	None
cmap	str	Matplotlib colormap. Defaults to 'hsv'.	'hsv'
dpi	int	Output DPI. Defaults to 200.	200
elev	float	3D elevation angle. Defaults to 20.	20
azim	float	3D azimuth angle. Defaults to -60.	-60
lw	float	Line width. Defaults to 1.0.	1.0
axes	bool	Draw the axes and tick labels. Set to False for a clean render with all axes and text removed. Defaults to True.	True
crop	bool	Tighten the spatial limits to the marker extent and trim the surrounding whitespace, so the figure shows mostly the data. Defaults to False.	False
target_name	str	Output name. Defaults to None ("_pose_waterfall.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
**pose_kwargs		Forwarded to pose() if keypoints have to be computed.	{}

Returns:

Name	Type	Description
MgFigure	'MgFigure'	the 3D waterfall figure, or None if there are too few frames.

Source code in musicalgestures/_pose.py

def mg_pose_waterfall(self, style: str = 'trajectories', n_samples: int = 40, markers: list | None = None, color_by: str | None = None,
                      cmap: str = 'hsv', dpi: int = 200, elev: float = 20, azim: float = -60, lw: float = 1.0, axes: bool = True, crop: bool = False,
                      target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure":
    """
    Render a 3D spatio-temporal waterfall of the pose, cascading along the time (depth) axis —
    a pose-based counterpart to ``silhouette_waterfall()``. Uses cached pose keypoints from a
    previous ``pose()`` call when available; otherwise it runs pose estimation first (extra
    keyword arguments such as ``model``/``device``/``downsampling_factor`` are forwarded to
    ``pose()``).

    Args:
        style (str, optional): What to draw. ``'trajectories'`` (default) draws each marker's
            continuous path through (x, time, y); ``'markers'`` scatters the markers at
            ``n_samples`` time slices; ``'skeleton'`` draws the skeleton joint lines at each
            time slice; ``'both'`` draws markers + skeleton.
        n_samples (int, optional): Number of time slices for the marker/skeleton styles.
            Defaults to 40 (ignored for ``'trajectories'``).
        markers (list, optional): Subset of marker names or indices to draw. Defaults to all.
        color_by (str, optional): ``'marker'`` or ``'time'``. Defaults to None ("auto"):
            'marker' for trajectories, 'time' for the slice styles.
        cmap (str, optional): Matplotlib colormap. Defaults to 'hsv'.
        dpi (int, optional): Output DPI. Defaults to 200.
        elev (float, optional): 3D elevation angle. Defaults to 20.
        azim (float, optional): 3D azimuth angle. Defaults to -60.
        lw (float, optional): Line width. Defaults to 1.0.
        axes (bool, optional): Draw the axes and tick labels. Set to False for a clean render
            with all axes and text removed. Defaults to True.
        crop (bool, optional): Tighten the spatial limits to the marker extent and trim the
            surrounding whitespace, so the figure shows mostly the data. Defaults to False.
        target_name (str, optional): Output name. Defaults to None ("_pose_waterfall.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        **pose_kwargs: Forwarded to ``pose()`` if keypoints have to be computed.

    Returns:
        MgFigure: the 3D waterfall figure, or None if there are too few frames.
    """
    from musicalgestures._pose_visualize import render_pose_waterfall

    _ensure_pose_keypoints(self, **pose_kwargs)

    c = self._pose_keypoints
    if target_name is None:
        target_name = c['of'] + '_pose_waterfall.png'
    else:
        target_name = os.path.splitext(target_name)[0] + '.png'

    mgf = render_pose_waterfall(
        c['data'], c['names'], c['width'], c['height'], c['fps'], target_name,
        overwrite=overwrite, style=style, connections=c.get('connections'),
        n_samples=n_samples, markers=markers, color_by=color_by, cmap=cmap,
        dpi=dpi, elev=elev, azim=azim, lw=lw, axes=axes, crop=crop)
    self.pose_waterfall_figure = mgf
    return mgf

pose_segments

pose_segments(segments=None, n_bins=36, cmap='viridis', dpi=200, ncols=6, target_name=None, overwrite=True, **pose_kwargs)

Circular (polar) motion plots and statistics for each body segment.

A segment is the bone between two connected joints (e.g. shoulder–elbow). For every segment this computes its per-frame orientation angle and draws a polar rose histogram of the angle distribution with the mean-direction resultant vector, annotated with circular statistics (mean angle, resultant length R, and range of motion). A CSV of the per-segment statistics — mean angle, R, circular std, range of motion, and mean angular speed — is saved alongside the image. Uses cached pose keypoints from a previous pose() call when available; otherwise it runs pose estimation first (model/device/… are forwarded to pose()).

Parameters:

Name	Type	Description	Default
segments	list	Subset of connections as (a, b) joint-index tuples. Defaults to all skeleton connections.	None
n_bins	int	Number of angular bins per rose. Defaults to 36 (10° bins).	36
cmap	str	Matplotlib colormap for the bars. Defaults to 'viridis'.	'viridis'
dpi	int	Output DPI. Defaults to 200.	200
ncols	int	Columns in the subplot grid. Defaults to 6.	6
target_name	str	Output name. Defaults to None ("_pose_segments.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
**pose_kwargs		Forwarded to pose() if keypoints have to be computed.	{}

Returns:

Name	Type	Description
MgFigure	'MgFigure'	the grid of circular plots (per-segment stats in .data['stats']), or None.

Source code in musicalgestures/_pose.py

def mg_pose_segments(self, segments: list | None = None, n_bins: int = 36, cmap: str = 'viridis', dpi: int = 200, ncols: int = 6,
                     target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure":
    """
    Circular (polar) motion plots and statistics for each body segment.

    A *segment* is the bone between two connected joints (e.g. shoulder–elbow). For every segment
    this computes its per-frame orientation angle and draws a polar rose histogram of the angle
    distribution with the mean-direction resultant vector, annotated with circular statistics
    (mean angle, resultant length R, and range of motion). A CSV of the per-segment statistics —
    mean angle, R, circular std, range of motion, and mean angular speed — is saved alongside the
    image. Uses cached pose keypoints from a previous ``pose()`` call when available; otherwise it
    runs pose estimation first (``model``/``device``/… are forwarded to ``pose()``).

    Args:
        segments (list, optional): Subset of connections as ``(a, b)`` joint-index tuples.
            Defaults to all skeleton connections.
        n_bins (int, optional): Number of angular bins per rose. Defaults to 36 (10° bins).
        cmap (str, optional): Matplotlib colormap for the bars. Defaults to 'viridis'.
        dpi (int, optional): Output DPI. Defaults to 200.
        ncols (int, optional): Columns in the subplot grid. Defaults to 6.
        target_name (str, optional): Output name. Defaults to None ("_pose_segments.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        **pose_kwargs: Forwarded to ``pose()`` if keypoints have to be computed.

    Returns:
        MgFigure: the grid of circular plots (per-segment stats in ``.data['stats']``), or None.
    """
    from musicalgestures._pose_visualize import render_segment_circular

    _ensure_pose_keypoints(self, **pose_kwargs)

    c = self._pose_keypoints
    if target_name is None:
        target_name = c['of'] + '_pose_segments.png'
    else:
        target_name = os.path.splitext(target_name)[0] + '.png'

    mgf = render_segment_circular(
        c['data'], c['names'], c.get('connections'), c['width'], c['height'], c['fps'],
        target_name, overwrite=overwrite, segments=segments, n_bins=n_bins, cmap=cmap,
        dpi=dpi, ncols=ncols)
    self.pose_segments_figure = mgf
    return mgf

pose_center

pose_center(save_data=True, dpi=200, target_name=None, overwrite=True, **pose_kwargs)

Centre the pose data on its global centroid — a 2D port of the MoCap Toolbox mccenter.

A single offset per coordinate (the mean of the per-marker temporal means, missing detections ignored) is subtracted from every marker so the overall spatiotemporal centroid sits at the origin (0, 0). This removes the performer's absolute position in the frame, leaving relative posture/movement — useful before comparing or further analysing trajectories. Plots the centred marker trajectories and (by default) saves a CSV of the centred coordinates. Uses cached pose keypoints when available, otherwise runs pose() first (**pose_kwargs are forwarded).

Parameters:

Name	Type	Description	Default
save_data	bool	Save a CSV of the centred coordinates. Defaults to True.	True
dpi	int	Output DPI. Defaults to 200.	200
target_name	str	Output name. Defaults to None ("_pose_centered.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
**pose_kwargs		Forwarded to pose() if keypoints have to be computed.	{}

Returns:

Name	Type	Description
MgFigure	'MgFigure'	the centred-trajectories figure; .data['coords'] holds the (T, n, 2) centred
	'MgFigure'	coordinates and .data['offset'] the removed centroid. None if there are too few frames.

Source code in musicalgestures/_pose.py

def mg_pose_center(self, save_data: bool = True, dpi: int = 200, target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure":
    """
    Centre the pose data on its global centroid — a 2D port of the MoCap Toolbox ``mccenter``.

    A single offset per coordinate (the mean of the per-marker temporal means, missing detections
    ignored) is subtracted from every marker so the overall spatiotemporal centroid sits at the
    origin (0, 0). This removes the performer's absolute position in the frame, leaving relative
    posture/movement — useful before comparing or further analysing trajectories. Plots the centred
    marker trajectories and (by default) saves a CSV of the centred coordinates. Uses cached pose
    keypoints when available, otherwise runs ``pose()`` first (``**pose_kwargs`` are forwarded).

    Args:
        save_data (bool, optional): Save a CSV of the centred coordinates. Defaults to True.
        dpi (int, optional): Output DPI. Defaults to 200.
        target_name (str, optional): Output name. Defaults to None ("_pose_centered.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        **pose_kwargs: Forwarded to ``pose()`` if keypoints have to be computed.

    Returns:
        MgFigure: the centred-trajectories figure; ``.data['coords']`` holds the (T, n, 2) centred
        coordinates and ``.data['offset']`` the removed centroid. None if there are too few frames.
    """
    from musicalgestures._pose_visualize import render_pose_center

    _ensure_pose_keypoints(self, **pose_kwargs)

    c = self._pose_keypoints
    if target_name is None:
        target_name = c['of'] + '_pose_centered.png'
    else:
        target_name = os.path.splitext(target_name)[0] + '.png'

    mgf = render_pose_center(c['data'], c['names'], c['width'], c['height'], target_name,
                             overwrite=overwrite, dpi=dpi)
    if mgf is not None and save_data:
        try:
            import pandas as pd
            coords = mgf.data['coords']            # (T, n, 2)
            times = mgf.data['times']
            cols = {'Time': np.round(times * 1000.0, 3)}
            for i, name in enumerate(c['names']):
                cols[f'{name} X'] = coords[:, i, 0]
                cols[f'{name} Y'] = coords[:, i, 1]
            pd.DataFrame(cols).to_csv(os.path.splitext(target_name)[0] + '.csv', index=False)
        except Exception as e:
            print(f'Warning: could not save CSV: {e}')
    self.pose_centered_figure = mgf
    return mgf

pose_distance

pose_distance(dpi=200, target_name=None, overwrite=True, **pose_kwargs)

Per-marker distance travelled and the average across markers — a 2D port of the MoCap Toolbox mccumdist.

Sums each marker's frame-to-frame Euclidean displacement (in pixels) and accumulates it over time. The figure shows the per-marker cumulative-distance curves and a ranked bar chart of the total distance per marker with the across-marker average marked; a CSV of the totals (plus the average) is saved. Uses cached pose keypoints when available, otherwise runs pose() first (**pose_kwargs are forwarded).

Parameters:

Name	Type	Description	Default
dpi	int	Output DPI. Defaults to 200.	200
target_name	str	Output name. Defaults to None ("_pose_distance.png").	None
overwrite	bool	Overwrite or auto-increment the filename. Defaults to True.	True
**pose_kwargs		Forwarded to pose() if keypoints have to be computed.	{}

Returns:

Name	Type	Description
MgFigure	'MgFigure'	.data['total'] (per-marker totals), .data['average'] (mean total), and
	'MgFigure'	.data['cumulative'] (per-marker cumulative curves). None if there are too few frames.

Source code in musicalgestures/_pose.py

def mg_pose_distance(self, dpi: int = 200, target_name: str | None = None, overwrite: bool = True, **pose_kwargs) -> "MgFigure":
    """
    Per-marker distance travelled and the average across markers — a 2D port of the MoCap Toolbox
    ``mccumdist``.

    Sums each marker's frame-to-frame Euclidean displacement (in pixels) and accumulates it over
    time. The figure shows the per-marker cumulative-distance curves and a ranked bar chart of the
    total distance per marker with the across-marker average marked; a CSV of the totals (plus the
    average) is saved. Uses cached pose keypoints when available, otherwise runs ``pose()`` first
    (``**pose_kwargs`` are forwarded).

    Args:
        dpi (int, optional): Output DPI. Defaults to 200.
        target_name (str, optional): Output name. Defaults to None ("_pose_distance.png").
        overwrite (bool, optional): Overwrite or auto-increment the filename. Defaults to True.
        **pose_kwargs: Forwarded to ``pose()`` if keypoints have to be computed.

    Returns:
        MgFigure: ``.data['total']`` (per-marker totals), ``.data['average']`` (mean total), and
        ``.data['cumulative']`` (per-marker cumulative curves). None if there are too few frames.
    """
    from musicalgestures._pose_visualize import render_pose_distance

    _ensure_pose_keypoints(self, **pose_kwargs)

    c = self._pose_keypoints
    if target_name is None:
        target_name = c['of'] + '_pose_distance.png'
    else:
        target_name = os.path.splitext(target_name)[0] + '.png'

    mgf = render_pose_distance(c['data'], c['names'], c['width'], c['height'], c['fps'],
                               target_name, overwrite=overwrite, dpi=dpi)
    self.pose_distance_figure = mgf
    return mgf

__repr__

__repr__()

Source code in musicalgestures/_video.py

def __repr__(self) -> str:
    w, h = getattr(self, 'width', None), getattr(self, 'height', None)
    size = f"{w}x{h}" if w and h else "?x?"
    fps = getattr(self, 'fps', None)
    fps_str = f"{fps:g}fps" if fps else "?fps"
    frames = getattr(self, 'length', None)
    frames_str = f"{int(frames)} frames" if frames else "? frames"
    return (f"MgVideo('{self.filename}', {frames_str}, {fps_str}, {size}, "
            f"audio={getattr(self, 'has_audio', None)})")

average

average(**kwargs)

Backward compatibility alias for blend(component_mode='average'). Creates an average image of all frames in the video.

Parameters:

Name	Type	Description	Default
**kwargs		Additional arguments passed to blend method. Note: 'normalize' parameter is accepted for backward compatibility but ignored.	{}

Returns:

Name	Type	Description
MgImage		A new MgImage pointing to the output average image file.

Source code in musicalgestures/_video.py

def average(self, **kwargs):
    """
    Backward compatibility alias for blend(component_mode='average').
    Creates an average image of all frames in the video.

    Args:
        **kwargs: Additional arguments passed to blend method.
                 Note: 'normalize' parameter is accepted for backward compatibility but ignored.

    Returns:
        MgImage: A new MgImage pointing to the output average image file.
    """
    # Strip parameters that were documented in older API versions but aren't
    # supported by the underlying blend implementation.
    filtered_kwargs = {k: v for k, v in kwargs.items() if k not in ('normalize', 'method')}
    return self.blend(component_mode='average', **filtered_kwargs)

test_input

test_input()

Gives feedback to user if initialization from input went wrong.

Source code in musicalgestures/_video.py

def test_input(self):
    """Gives feedback to user if initialization from input went wrong."""
    mg_input_test(
        self.filename,
        self.array,
        self.fps,
        self.filtertype,
        self.threshold,
        self.starttime,
        self.endtime,
        self.blur,
        self.skip,
        self.frames,
    )

get_video

get_video()

Creates a video attribute to the Musical Gestures object with the given correct settings.

NB: For an MgVideo, self.length is the number of frames (from get_framecount), whereas for MgAudio self.length is the duration in seconds. To get the video duration in seconds use self.length / self.fps.

Source code in musicalgestures/_video.py

def get_video(self):
    """Creates a video attribute to the Musical Gestures object with the given correct settings.

    NB: For an ``MgVideo``, ``self.length`` is the number of **frames** (from
    ``get_framecount``), whereas for ``MgAudio`` ``self.length`` is the duration in
    **seconds**. To get the video duration in seconds use ``self.length / self.fps``.
    """
    # Bake any display-rotation flag into the pixels so every reader (FFmpeg pipe,
    # OpenCV, filters) agrees on orientation and no process comes out rotated.
    from musicalgestures._utils import normalize_rotation
    oriented = normalize_rotation(self.filename)
    if oriented != self.filename:
        self.filename = oriented
        self.of, self.fex = os.path.splitext(self.filename)

    (
        self.length,
        self.width,
        self.height,
        self.fps,
        self.endtime,
        self.of,
        self.fex,
        self.has_audio,
    ) = mg_videoreader(
        filename=self.filename,
        starttime=self.starttime,
        endtime=self.endtime,
        skip=self.skip,
        frames=self.frames,
        rotate=self.rotate,
        contrast=self.contrast,
        brightness=self.brightness,
        crop=self.crop,
        color=self.color,
        returned_by_process=self.returned_by_process,
        keep_all=self.keep_all,
    )

    # Convert eventual low-resolution video or image
    video_formats = [
        ".avi",
        ".mp4",
        ".mov",
        ".mkv",
        ".mpg",
        ".mpeg",
        ".webm",
        ".ogg",
        ".ts",
        ".wmv",
        ".3gp",
        ".360",
    ]
    if self.fex not in video_formats:
        # Check if it is an image file
        if get_framecount(self.filename) == 1:
            image_formats = [
                ".gif",
                ".jpeg",
                ".jpg",
                ".jfif",
                ".pjpeg",
                ".png",
                ".svg",
                ".webp",
                ".avif",
                ".apng",
            ]
            if self.fex not in image_formats:
                # Create one converted version and register it to the MgVideo
                filename = convert(
                    self.of + self.fex, self.of + self.fex + ".png", overwrite=True
                )
                # point of and fex to the png version
                self.of, self.fex = os.path.splitext(filename)
            else:
                # update filename after the processes
                self.filename = self.of + self.fex
        else:
            # Create one converted version and register it to the MgVideo
            filename = convert_to_mp4(self.of + self.fex, overwrite=True)
            # point of and fex to the mp4 version
            self.of, self.fex = os.path.splitext(filename)
    else:
        # Update filename after the processes
        self.filename = self.of + self.fex

    # Check if there is audio in the video file
    if self.has_audio:
        self.audio = MgAudio(self.filename, self.sr, self.n_fft, self.hop_length)
    else:
        self.audio = None

numpy

numpy()

Read all video frames into a numpy array using FFmpeg.

Returns:

Name	Type	Description
tuple		A tuple (array, fps) where array is a numpy.ndarray of shape (N, H, W, 3) in BGR format (uint8) containing all N frames, and fps is the frame rate of the video.

Source code in musicalgestures/_video.py

def numpy(self):
    """
    Read all video frames into a numpy array using FFmpeg.

    Returns:
        tuple: A tuple ``(array, fps)`` where ``array`` is a ``numpy.ndarray``
            of shape ``(N, H, W, 3)`` in BGR format (uint8) containing all N
            frames, and ``fps`` is the frame rate of the video.
    """
    # Define ffmpeg command and load all the video frames in memory
    cmd = ["ffmpeg", "-y", "-i", self.filename]
    process = ffmpeg_cmd(cmd, total_time=self.length, pipe="load")
    # Convert bytes to numpy array
    array = np.frombuffer(process.stdout, dtype=np.uint8).reshape(
        -1, self.height, self.width, 3
    )

    return array, self.fps

from_numpy

from_numpy(array, fps, target_name=None)

Writes a numpy array of video frames to a video file using FFmpeg.

After writing, updates self.filename, self.of, and self.fex to reflect the actual output path so that subsequent operations on this object refer to the newly created file.

Parameters:

Name	Type	Description	Default
array	ndarray	Video frames array with shape (N, H, W, 3) in BGR format.	required
fps	float	Frames per second for the output video.	required
target_name	str	Full path for the output file. If None, uses self.path/self.filename (or just self.filename if path is None). Defaults to None.	None

Source code in musicalgestures/_video.py

def from_numpy(self, array: np.ndarray, fps: float, target_name: str | None = None) -> None:
    """
    Writes a numpy array of video frames to a video file using FFmpeg.

    After writing, updates ``self.filename``, ``self.of``, and ``self.fex`` to
    reflect the actual output path so that subsequent operations on this object
    refer to the newly created file.

    Args:
        array (np.ndarray): Video frames array with shape (N, H, W, 3) in BGR format.
        fps (float): Frames per second for the output video.
        target_name (str, optional): Full path for the output file. If None, uses
            ``self.path/self.filename`` (or just ``self.filename`` if path is None).
            Defaults to None.
    """
    if target_name is not None:
        write_path = os.path.splitext(target_name)[0] + self.fex
    elif self.path is not None:
        write_path = os.path.join(self.path, self.filename)
    else:
        write_path = self.filename

    process = None
    for frame in array:
        if process is None:
            cmd = [
                "ffmpeg",
                "-y",
                "-s",
                "{}x{}".format(frame.shape[1], frame.shape[0]),
                "-r",
                str(fps),
                "-f",
                "rawvideo",
                "-pix_fmt",
                "bgr24",
                "-vcodec",
                "rawvideo",
                "-i",
                "-",
                "-vcodec",
                "libx264",
                "-pix_fmt",
                "yuv420p",
                write_path,
            ]
            process = ffmpeg_cmd(cmd, total_time=array.shape[0], pipe="write")
        process.stdin.write(frame.astype(np.uint8))
    process.stdin.close()
    process.wait()

    # Update self.filename to the actual written path so that get_video() can find the file
    self.filename = write_path
    self.of, self.fex = os.path.splitext(write_path)

extract_frame

extract_frame(**kwargs)

Extracts a frame from the video at a given time. see _utils.extract_frame for details.

Other Parameters:

Name	Type	Description
frame	int	The frame number to extract.
time	str	The time in HH:MM:ss.ms where to extract the frame from.
target_name	str	The name for the output file. If None, the name will be FRAME.. Defaults to None.
overwrite	bool	Whether to allow overwriting existing files or to automatically increment target filename to avoid overwriting. Defaults to True.

Returns:

Name	Type	Description
MgImage		An MgImage object referring to the extracted frame.

Source code in musicalgestures/_video.py

def extract_frame(self, **kwargs):
    """
    Extracts a frame from the video at a given time.
    see _utils.extract_frame for details.

    Keyword Args:
        frame (int): The frame number to extract.
        time (str): The time in HH:MM:ss.ms where to extract the frame from.
        target_name (str, optional): The name for the output file. If None, the name will be <input name>FRAME<frame number>.<file extension>. Defaults to None.
        overwrite (bool, optional): Whether to allow overwriting existing files or to automatically increment target filename to avoid overwriting. Defaults to True.

    Returns:
        MgImage: An MgImage object referring to the extracted frame.
    """
    return MgImage(extract_frame(self.filename, **kwargs))