Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex

Micheli, Cristiano; Schepers, Inga M.; Özker, Müge; Yoshor, Daniel; Beauchamp, Michael S.; Rieger, Jochem W.

doi:10.1111/ejn.13992

Cited by 27 publications

(39 citation statements)

References 71 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our data extend those findings to audiovisual speech. Of note, although our results highlight right hemispheric regions, it has been amply demonstrated that the left hemisphere, and especially the left superior temporal cortex, also participates in audiovisual speech integration ( 10 , 41 – 43 ).…”

Section: Discussioncontrasting

confidence: 54%

The phase of cortical oscillations determines the perceptual fate of visual cues in naturalistic audiovisual speech

2020

View full text Add to dashboard Cite

When we see our interlocutor, our brain seamlessly extracts visual cues from their face and processes them along with the sound of their voice, making speech an intrinsically multimodal signal. Visual cues are especially important in noisy environments, when the auditory signal is less reliable. Neuronal oscillations might be involved in the cortical processing of audiovisual speech by selecting which sensory channel contributes more to perception. To test this, we designed computer-generated naturalistic audiovisual speech stimuli where one mismatched phoneme-viseme pair in a key word of sentences created bistable perception. Neurophysiological recordings (high-density scalp and intracranial electroencephalography) revealed that the precise phase angle of theta-band oscillations in posterior temporal and occipital cortex of the right hemisphere was crucial to select whether the auditory or the visual speech cue drove perception. We demonstrate that the phase of cortical oscillations acts as an instrument for sensory selection in audiovisual speech processing.

show abstract

Section: Discussioncontrasting

confidence: 54%

The phase of cortical oscillations determines the perceptual fate of visual cues in naturalistic audiovisual speech

2020

View full text Add to dashboard Cite

show abstract

“…This phase alignment in turn determines systematic, stimulus-locked variations in neuronal activity, as indexed by fluctuations in broadband high-frequency activity. It was further shown that visual speech gestures enhance intelligibility by facilitating auditory cortical entrainment to the speech stream (Crosse, Butler and Lalor, 2015;Perrodin et al, 2015;Park et al, 2016Park et al, , 2018Di Liberto et al, 2018;Micheli et al, 2018). Here, we used iEEG recordings for a more direct examination of the neurophysiological mechanisms underlying visual enhancement of auditory cortical speech processing.…”

Section: Discussionmentioning

confidence: 99%

Phase resetting in human auditory cortex to visual speech

Mégevand

Mercier

Groppe

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Natural conversation is multisensory: when we can see the speaker's face, visual speech cues influence our perception of what is being said. The neuronal basis of this phenomenon remains unclear, though there is indication that neuronal oscillations-ongoing excitability fluctuations of neuronal populations in the brain-represent a potential mechanism. Investigating this question with intracranial recordings in humans, we show that some sites in auditory cortex track the temporal dynamics of unisensory visual speech using the phase of their slow oscillations and phase-related modulations in neuronal activity. This effect is asymmetric, as we find much less detectable tracking of auditory speech by visual cortex. Auditory cortex thus builds a representation of the speech stream's envelope based on visual speech alone, at least in part by resetting the phase of its ongoing oscillations. Phase reset amplifies the representation of the speech stream and organizes the information contained in neuronal activity patterns.Keywords: neuronal oscillations, crossmodal stimuli, intracranial EEG, intertrial phase coherence, broadband high-frequency activity, phase-amplitude coupling reconstruction; Bahar Khalighinejad for help with stimulus reconstruction. Part of the computations for this work were performed at the University of Geneva on the Baobab cluster.

show abstract

“…For example, electrophysiological research in nonhuman primates shows that neural activity in primary auditory cortex is modulated by multisensory cues, including from motor/haptic (Lakatos, Chen, O'Connell, Mills, & Schroeder, 2007) and visual (Lakatos, Karmos, Mehta, Ulbert, & Schroeder, 2008) inputs. In humans, regions of posterior STS also represent visual amplitude envelope information (Micheli et al, 2018). Non-auditory information may arrive at primary auditory cortex through nonspecific thalamic pathways (Schroeder, Lakatos, Kajikawa, Partan, & Puce, 2008) and/or cortico-cortical connections from visual to auditory cortex (Arnal & Giraud, 2012).…”

Section: Early Integration Models Of Audiovisual Integrationmentioning

confidence: 99%

Audiovisual speech perception: Moving beyond McGurk

Engen¹,

Dey²,

Sommers³

et al. 2019

Preprint

View full text Add to dashboard Cite

Although listeners use both auditory and visual cues during speech perception, the cognitive and neural bases for their integration remain a matter of debate. One common approach to measuring multisensory integration is to use McGurk tasks, in which discrepant auditory and visual cues produce auditory percepts that differ from those based solely on unimodal input. Not all listeners show the same degree of susceptibility to the McGurk illusion, and these individual differences in susceptibility are frequently used as a measure of audiovisual integration ability. However, despite their popularity, we argue that McGurk tasks are ill-suited for studying the kind of multisensory speech perception that occurs in real life: McGurk stimuli are often based on isolated syllables (which are rare in conversations) and necessarily rely on audiovisual incongruence that does not occur naturally. Furthermore, recent data show that susceptibility on McGurk tasks does not correlate with performance during natural audiovisual speech perception. Although the McGurk effect is a fascinating illusion, truly understanding the combined use of auditory and visual information during speech perception requires tasks that more closely resemble everyday communication.

show abstract

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex

Cited by 27 publications

References 71 publications

The phase of cortical oscillations determines the perceptual fate of visual cues in naturalistic audiovisual speech

The phase of cortical oscillations determines the perceptual fate of visual cues in naturalistic audiovisual speech

Phase resetting in human auditory cortex to visual speech

Audiovisual speech perception: Moving beyond McGurk

Contact Info

Product

Resources

About