Audiovisuality¶
Physics¶
- Light: Light is electromagnetic radiation within the visible spectrum, enabling vision and the perception of color, shape, and movement. It is fundamental to how we interpret our surroundings. Light behaves both as a wave and as a particle (photon), and its properties—such as wavelength and intensity—determine how we perceive brightness and color.
- Sound: Sound consists of mechanical vibrations that travel through a medium (such as air or water) and are perceived by the auditory system. It underpins communication, music, and environmental awareness. Sound is characterized by properties like frequency (pitch), amplitude (loudness), and timbre (quality).
Representation¶
- Audio: Audio encompasses the capture, transmission, and reproduction of sound. It is essential in media, communication, and interactive technologies, allowing for the sharing and analysis of auditory information. Audio signals can be analog or digital, and are often processed for clarity, compression, or creative effect.
- Video: Video involves recording, processing, and displaying sequences of moving images. It is a versatile medium for conveying information, storytelling, education, and entertainment. Video signals can be analog or digital, and are defined by parameters such as frame rate, resolution, and color depth.
These fundamental physical concepts and their representations form the basis for understanding audiovisual systems and technologies, as well as the ways in which humans perceive and interact with their environment.
Auditory-visual¶
- Auditory-visual: Auditory-visual integration refers to the brain’s ability to combine auditory and visual information to enhance perception and understanding. This phenomenon is essential in activities like speech comprehension and multimedia experiences. Learn more on Wikipedia.
Multimodal¶
- Multimodal: Multimodal perception involves the integration of information from multiple sensory modalities, such as sight, sound, and touch, to create a cohesive understanding of the environment. Learn more on Wikipedia.
Crossmodal¶
- McGurk effect: The McGurk effect is a perceptual phenomenon where conflicting auditory and visual stimuli result in a third, distinct perception. It highlights the complex interplay between sensory modalities. Learn more on Wikipedia.
Psychology¶
Gaze¶
Gaze refers to the direction in which a person is looking, often used as an indicator of attention and focus. It plays a crucial role in understanding human behavior, communication, and cognitive processes. Gaze tracking is commonly used in psychological studies to analyze visual attention and social interactions. Learn more on Wikipedia.
Pupillometry¶
Pupillometry is the measurement of pupil size and reactivity, often used to study cognitive and emotional processes. Changes in pupil size can indicate arousal, attention, and mental effort. This technique is widely applied in psychology, neuroscience, and human-computer interaction research. Learn more on Wikipedia.
Technology¶
Eye-Trackers¶
Eye-trackers are devices used to measure eye positions and movements. They are widely used in research fields such as psychology, neuroscience, marketing, and human-computer interaction. Eye-trackers can be categorized into two main types: mobile and stationary.
Mobile Eye-Trackers¶
Mobile eye-trackers are wearable devices that allow for the tracking of eye movements in real-world environments. These devices are often used in studies that require participants to move freely, such as sports performance analysis, usability testing, and outdoor experiments. Learn more on Wikipedia.
Figure 1: Example of mobile eye-tracking glasses. Image credit: Wikimedia Commons.
Stationary Eye-Trackers¶
Stationary eye-trackers are fixed devices typically used in controlled laboratory settings. They are often mounted on a desk or integrated into a monitor and are used for tasks such as reading studies, visual search experiments, and website usability testing. Learn more on Wikipedia.
Figure 2: Example of stationary eye-tracking software. Image credit: Wikimedia Commons.