MEG Activity in Visual and Auditory Cortices Represents Acoustic Speech-Related Information during Silent Lip Reading

Bröhl, Felix; Keitel, Anne; Kayser, Christoph

doi:10.1523/eneuro.0209-22.2022

Cited by 13 publications

(8 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our current study demonstrates that visual speech information can be decoded in the auditory region, aligning with a growing body of research, including the findings of Brohl et al (54), which underscores the multisensory nature of speech processing. While both studies underscore auditory cortex’s role in integrating visual speech, our approach, utilizing a combination of fMRI and intracranial EEG, provides unique spatial precision.…”

Section: Discussionsupporting

confidence: 90%

Auditory cortex encodes lipreading information through spatially distributed activity

Ganesan

Cao

Demidenko

et al. 2022

Preprint

View full text Add to dashboard Cite

Face-to-face communication improves the quality and accuracy of heard speech, particularly in noisy environments. Silent lipreading modulates activity in auditory regions, which has been hypothesized to reflect the transformation and encoding of multiple forms of visual speech information used to support hearing processes. Evidence suggests visual timing information as one such signal encoded in auditory areas: seeing when a speaker's lips come together between words can help listeners parse word-level boundaries. However, it remains unclear how lipreading alters activity in the auditory system to improve speech perception at the single word-level. Using fMRI and intracranial electrodes in patients, here we show that silently lipread words can be classified from neural activity in auditory areas based on distributed spatial information. Lipread words evoked similar representations to the corresponding heard words, consistent with the prediction that automatic lipreading refines the tuning of auditory representations. Similar to heard words, lipread words varied in the distinctiveness of their neural representations in auditory cortex: e.g., the lipread words DIG and GIG evoked more similar neural activity in auditory cortex relative to the more perceptually distinct word FIG, suggesting that lipreading activity reflects probabilistic distributions as opposed to the unique identity of the lipread word. Notably, while visual speech has both excitatory and suppressive effects on auditory firing rates, classification was observed in both neural populations, consistent with the prediction that lipreading contributes to phoneme population tuning by both activating the corresponding representation and suppressing incorrect phonemic representations. These results support a model in which the auditory system combines the joint neural distributions evoked by heard and lipread words to generate a more precise estimate of what was said, particularly during noisy speech.

show abstract

Section: Discussionsupporting

confidence: 90%

Auditory cortex encodes lipreading information through spatially distributed activity

Ganesan

Cao

Demidenko

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Neural speech tracking is widely used to study the neural processing of continuous speech, though primarily with audio-only stimuli (Brodbeck, Hong, et al, 2018;Chalas et al, 2022;Di Liberto et al, 2015;Keitel et al, 2018). Recent studies have used audiovisual speech settings, but without directly modeling the visual speech features Golumbic et al, 2013) or not incorporating their temporal dynamics due to the use of frequency-based methods (Aller et al, 2022;Bröhl et al, 2022;Park et al, 2016). Here, we show, for the first time, the temporal dynamics and cortical origins of TRFs obtained from lip movements in an audiovisual setting with one or two speakers.…”

Section: Discussionmentioning

confidence: 99%

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Reisinger

Gillis

Suess

et al. 2023

Preprint

View full text Add to dashboard Cite

Observing lip movements of a speaker is known to facilitate speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows enhanced processing of audiovisual stimuli. However, the interindividual variability of this visual benefit and its consequences on behavior are unknown. Here, we analyzed source-localized magnetoencephalographic (MEG) responses from normal-hearing participants listening to audiovisual speech with or without an additional distractor speaker. Using temporal response functions (TRFs), we show that neural responses to lip movements are, in general, enhanced when speech is challenging. After conducting a crucial control for speech acoustics, we show that lip movements effectively contribute to higher neural speech tracking, particularly when a distractor speaker is present. However, the extent of this visual benefit varied greatly among participants. Probing the behavioral relevance, we show that individuals who benefit more from lip movement information in terms of neural speech tracking, show a stronger drop in performance and an increase in perceived difficulty when the mouth is occluded by a surgical face mask. By contrast, no effect was found when the mouth was not occluded. We provide novel insights on how the benefit of lip movements in terms of neural speech tracking varies among individuals. Furthermore, we reveal its behavioral relevance by demonstrating negative consequences for behavior when visual speech is absent. Our results also offer potential implications for future objective assessments of audiovisual speech perception.

show abstract

“…MI was then computed between the brain signal and the 3000 shuffled envelope/surprise signals. The group level mean was then tested against the 95th percentile of the random group mean distribution, essentially implementing a one-sided randomisation test at p < .05 (Brohl et al ., 2022).…”

Section: Methodsmentioning

confidence: 99%

Cortical and behavioural tracking of rhythm in music: Effects of pitch predictability, enjoyment, and expertise

Keitel,

Pelofi,

Guan

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Cortical tracking of stimulus features (such as the envelope) is a crucial tractable neural mechanism, allowing us to investigate how we process continuous music. We here tested whether cortical and behavioural tracking of beat, typically related to rhythm processing, are modulated by pitch predictability. In two experiments (n=20, n=52), participants' ability to tap along to the beat of musical sequences was measured for tonal (high pitch predictability) and atonal (low pitch predictability) music. In Experiment 1, we additionally measured participants' EEG and analysed cortical tracking of the acoustic envelope and of pitch surprisal (using IDyOM). In both experiments, finger-tapping performance was better in the tonal than the atonal condition, indicating a positive effect of pitch predictability on behavioural rhythm processing. Neural data revealed that the acoustic envelope was tracked stronger while listening to atonal than tonal music, potentially reflecting listeners' violated pitch expectations. Our findings show that cortical envelope tracking, beyond reflecting musical rhythm processing, is modulated by pitch predictability (as well as musical expertise and enjoyment). Stronger cortical surprisal tracking was linked to overall worse envelope tracking, and worse finger-tapping performance for atonal music. Specifically, the low pitch predictability in atonal music seems to draw attentional resources resulting in a reduced ability to follow the rhythm behaviourally. Overall, cortical envelope and surprisal tracking were differentially related to behaviour in tonal and atonal music, likely reflecting differential processing under conditions of high and low predictability. Taken together, our results show diverse effects of pitch predictability on musical rhythm processing.

show abstract

MEG Activity in Visual and Auditory Cortices Represents Acoustic Speech-Related Information during Silent Lip Reading

Cited by 13 publications

References 79 publications

Auditory cortex encodes lipreading information through spatially distributed activity

Auditory cortex encodes lipreading information through spatially distributed activity

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

Cortical and behavioural tracking of rhythm in music: Effects of pitch predictability, enjoyment, and expertise

Contact Info

Product

Resources

About