Congruent Visual Speech Enhances Cortical Entrainment to Continuous Auditory Speech in Noise-Free Conditions

Abstract

Congruent audiovisual speech enhances our ability to comprehend a speaker, even in noise-free conditions. When incongruent auditory and visual information is presented concurrently, it can hinder a listener's perception and even cause him or her to perceive information that was not presented in either modality. Efforts to investigate the neural basis of these effects have often focused on the special case of discrete audiovisual syllables that are spatially and temporally congruent, with less work done on the case of natural, continuous speech. Recent electrophysiological studies have demonstrated that cortical response measures to continuous auditory speech can be easily obtained using multivariate analysis methods. Here, we apply such methods to the case of audiovisual speech and, importantly, present a novel framework for indexing multisensory integration in the context of continuous speech. Specifically, we examine how the temporal and contextual congruency of ongoing audiovisual speech affects the cortical encoding of the speech envelope in humans using electroencephalography. We demonstrate that the cortical representation of the speech envelope is enhanced by the presentation of congruent audiovisual speech in noise-free conditions. Furthermore, we show that this is likely attributable to the contribution of neural generators that are not particularly active during unimodal stimulation and that it is most prominent at the temporal scale corresponding to syllabic rate (2-6 Hz). Finally, our data suggest that neural entrainment to the speech envelope is inhibited when the auditory and visual streams are incongruent both temporally and contextually.

Prediction Errors but Not Sharpened Signals Simulate Multivoxel fMRI Patterns during Speech Perception

“…In contrast, enhanced representation of attended compared to unattended speech supports Sharpening mechanisms [84–87]. These findings could be reconciled by theories proposing that expectation (Prediction Error) and attention (Sharpening) operate in parallel, as suggested in some Predictive Coding theories [3,88].…”

The Multivariate Temporal Response Function (mTRF) Toolbox: A MATLAB Toolbox for Relating Neural Signals to Continuous Stimuli

“…The stimulus reconstruction method has previously been used to study both the visual and auditory system in various animal models (Bialek et al, 1991; Rieke et al, 1995; Stanley et al, 1999). More recently, it has been adopted for studying speech processing in the human brain using intracranial and non-invasive electrophysiology (Mesgarani et al, 2009; Pasley et al, 2012; Ding and Simon, 2013; Martin et al, 2014; Crosse et al, 2015a, Crosse et al, 2016; O'Sullivan et al, 2015). …”

Visual cortex entrains to sign language

“…Although entrainment is partly driven by bottom-up features of the stimulus (8-10), it also depends on top-down signals to auditory cortex from other brain areas. Auditory entrainment is strengthened when people see congruent visual and auditory information (11,12) and is modulated by attention (13) and by top-down signals from frontal cortex (4, 14).Cortical entrainment is proposed to perform a key role in speech comprehension, such as segmenting out syllables from a continuous speech stream (1, 2, 15) or optimizing perceptual sensitivity to rhythmic pulses of sound (5-7). However, the mechanisms driving entrainment to speech remain unclear.…”

“…Although entrainment is partly driven by bottom-up features of the stimulus (8)(9)(10), it also depends on top-down signals to auditory cortex from other brain areas. Auditory entrainment is strengthened when people see congruent visual and auditory information (11,12) and is modulated by attention (13) and by top-down signals from frontal cortex (4,14).…”

“…We consider two hypotheses. First, flexible entrainment to quasiperiodic rhythms may be specific to auditory perception (6); in visual perception, by contrast, cortical oscillations in the alpha band (8)(9)(10)(11)(12) Hz) may phase-lock only to consistent stimulus rhythms, without adjusting to variable stimulus rhythms (16). Second, lowfrequency cortical entrainment may be a general-purpose neural mechanism that helps optimize perception to time-varying stimuli regardless of the perceptual modality.…”