Substituting facial movements in singers changes the sounds of musical intervals

Abstract

Cross-modal integration is ubiquitous within perception and, in humans, the McGurk effect demonstrates that seeing a person articulating speech can change what we hear into a new auditory percept. It remains unclear whether cross-modal integration of sight and sound generalizes to other visible vocal articulations like those made by singers. We surmise that perceptual integrative effects should involve music deeply, since there is ample indeterminacy and variability in its auditory signals. We show that switching videos of sung musical intervals changes systematically the estimated distance between two notes of a musical interval so that pairing the video of a smaller sung interval to a relatively larger auditory led to compression effects on rated intervals, whereas the reverse led to a stretching effect. In addition, after seeing a visually switched video of an equally-tempered sung interval and then hearing the same interval played on the piano, the two intervals were judged often different though they differed only in instrument. These findings reveal spontaneous, cross-modal, integration of vocal sounds and clearly indicate that strong integration of sound and sight can occur beyond the articulations of natural speech.

Differential projections from the cochlear nucleus to the inferior colliculus in the mouse

“…Stereognosis, the ability to identify the 3D shape of an object manually or orally is linked to auditory discrimination ( Lewis and Kelly, 1974 ). The ventriloquism effect demonstrates that sound localization is influenced by vision ( Spence and Driver, 2000 ), and facial movements can affect the perception of speech ( McGurk and MacDonald, 1976 ) as well as non-speech stimuli ( Saldaña and Rosenblum, 1993 ; Laeng et al, 2021 ). There is also the sound-induced flash illusion where the observer misinterprets the number of visual flashes due to the simultaneous presentation of a different number of clicks ( Virsu et al, 2008 ).…”

Increased body movement equals better performance? Not always! Musical style determines motion degree perceived as optimal in music performance

“…Consequently, meaningful interactions rely on the successful encoding and decoding of expression, respectively, by the musicians and the audience (Juchniewicz, 2008 ; Leman, 2016 ). Musical expression, in turn, encompasses not only auditory elements such as rhythmic variations (Huberth et al, 2020 ; Repp, 1992 ) or timbre (Li & Timmers, 2020 ), but also non-auditory cues related to the facial and bodily actions of the performers (Davidson, 2012 ; Laeng et al, 2021 ; Nápoles et al, 2022 ; Weiss et al, 2018 ). Although body movement is not the ultimate goal in music performance, as it is in dance, there has been a growing interest in this dimension, considering that movements significantly affect observers’ opinion (Bugaj et al, 2019 ; Nápoles et al, 2022 ; Silveira, 2014 ; Trevor & Huron, 2018 ; Wapnick et al, 2004 ) and auditory perception (Juchniewicz, 2008 ; Nápoles et al, 2022 ; Schutz & Lipscomb, 2007 ; Wapnick et al, 2004 ).…”

Sound-producing actions in guitar performance of groove-based microrhythm

“…Some embodied perspectives emphasize that perception is an active process-something that we do-which is related to sense-making and is based on previous multimodal experiences (e.g., Noë, 2004;Shapiro, 2010;Varela et al, 2016;Van Der Schyff et al, 2018). So-called motor theories of perception further propose that sound perception includes not only processing of auditory input but also an understanding of what we believe causes a sound-that is, the listener takes into account the sound's source and/or the action that produced the sound (e.g., Berthoz, 2000;Cox, 2016;Godøy, 2003Godøy, , 2010Jensenius, 2007;Laeng et al, 2021;Liberman & Mattingly, 1985). Within this framework, sound perception is not only a matter of feature extraction based on the sound-signal but also includes knowledge of sound-source and sound-action relationships, based on previous embodied experiences of how sounds are produced.…”