The Natural Statistics of Audiovisual Speech

Abstract

Humans, like other animals, are exposed to a continuous stream of signals, which are dynamic, multimodal, extended, and time varying in nature. This complex input space must be transduced and sampled by our sensory systems and transmitted to the brain where it can guide the selection of appropriate actions. To simplify this process, it's been suggested that the brain exploits statistical regularities in the stimulus space. Tests of this idea have largely been confined to unimodal signals and natural scenes. One important class of multisensory signals for which a quantitative input space characterization is unavailable is human speech. We do not understand what signals our brain has to actively piece together from an audiovisual speech stream to arrive at a percept versus what is already embedded in the signal structure of the stream itself. In essence, we do not have a clear understanding of the natural statistics of audiovisual speech. In the present study, we identified the following major statistical features of audiovisual speech. First, we observed robust correlations and close temporal correspondence between the area of the mouth opening and the acoustic envelope. Second, we found the strongest correlation between the area of the mouth opening and vocal tract resonances. Third, we observed that both area of the mouth opening and the voice envelope are temporally modulated in the 2–7 Hz frequency range. Finally, we show that the timing of mouth movements relative to the onset of the voice is consistently between 100 and 300 ms. We interpret these data in the context of recent neural theories of speech which suggest that speech communication is a reciprocally coupled, multisensory event, whereby the outputs of the signaler are matched to the neural processes of the receiver.

A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula

“…For simplicity we focus here on a sensor‐level time‐domain analysis. Since previous work has shown speech entrainment mostly at lower frequencies, we extracted the wideband amplitude envelope of the speech stimulus [Chandrasekaran et al, 2009; Gross et al, 2013] and then low‐pass filtered with a 12 Hz cutoff (third order noncausal Butterworth). The MEG signal was obtained from a 248‐magnetometer whole‐head MEG system (MAGNES 3600 WH, 4D Neuroimaging).…”

Phonetic Recalibration in Audiovisual Speech

“…Firstly, it is appropriate to note that lipread input is bottom-up visual information that is usually perceived in synchrony with the auditory stream, although it actually often precedes the acoustic signal (Chandrasekaran et al, 2009). Lexical information, in contrast, exerts a top-down influence on auditory processing, and is argued to become important as the word is being recognized (e.g., Samuel & Pitt, 2003).…”

“…Given that in AV speech, lipread input usually precedes the auditory signal (e.g., Chandrasekaran, Trubanova, Stillittano, Caplier, & Ghazanfar, 2009), Stekelenburg and Vroomen (2007) argued that the visually-induced N1 modulations arise whenever the visual input precedes the audio, thus warning the listener about when the sound is going to occur. This was corroborated by similar results obtained with artificial AV stimuli in which anticipatory visual motion reliably predicted sound onset (Vroomen & Stekelenburg, 2010).…”

Transitions in neural oscillations reflect prediction errors generated in audiovisual speech

“…Our experimental paradigm capitalizes on a natural ~150-ms temporal lag between the onset of facial movements and vocal chord vibration that naturally occurs when we speak 9 . This lag allows the observer to synthesize phonological predictions before auditory onset on the basis of visual information.…”

“…Here, sensory prediction errors were induced experimentally by violating the congruence between visual and auditory information. We also exploited the specificity with which visual input predicts auditory input to create a gradient of perceived incongruence [8][9][10] . We thus induced graded prediction errors by varying audio-visual congruence and expected them to be manifest at the neural level in the interaction between the amount of predictive information conveyed by the visual input (predictiveness) and the validity of this information with respect to incoming auditory input (audio-visual congruence).…”