Differential Auditory and Visual Phase-Locking Are Observed during Audio-Visual Benefit and Silent Lip-Reading for Speech Perception

Abstract

Speech perception in noisy environments is enhanced by seeing facial movements of communication partners. However, the neural mechanisms by which audio and visual speech are combined are not fully understood. We explore MEG phase-locking to auditory and visual signals in MEG recordings from 14 human participants (6 females, 8 males) that reported words from single spoken sentences. We manipulated the acoustic clarity and visual speech signals such that critical speech information is present in auditory, visual, or both modalities. MEG coherence analysis revealed that both auditory and visual speech envelopes (auditory amplitude modulations and lip aperture changes) were phase-locked to 2-6 Hz brain responses in auditory and visual cortex, consistent with entrainment to syllable-rate components. Partial coherence analysis was used to separate neural responses to correlated audio-visual signals and showed non-zero phase-locking to auditory envelope in occipital cortex during audio-visual (AV) speech. Furthermore, phase-locking to auditory signals in visual cortex was enhanced for AV speech compared with audioonly speech that was matched for intelligibility. Conversely, auditory regions of the superior temporal gyrus did not show abovechance partial coherence with visual speech signals during AV conditions but did show partial coherence in visual-only conditions. Hence, visual speech enabled stronger phase-locking to auditory signals in visual areas, whereas phase-locking of visual speech in auditory regions only occurred during silent lip-reading. Differences in these cross-modal interactions between auditory and visual speech signals are interpreted in line with cross-modal predictive mechanisms during speech perception.

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

“…Brodbeck, Hong, et al, 2018;Kulasingham et al, 2020). The fROIs for the lip movements involved parietal and occipital regions, in line with previous studies that sourcelocalized the neural tracking of lip movements (Aller et al, 2022;Bourguignon et al, 2020;Hauswald et al, 2018). Similar to Park et al (2016), we also observed neural tracking of lip movements in temporal regions (see Figure S1), but with less involvement of the primary visual cortex and prominent only in the single speaker condition.…”

“…Analogous to behavioral findings in Aller et al (2022), the benefit of lip movements showed high interindividual variability (see Figure 2C) and followed a bimodal distribution. Some individuals benefited massively from lip movements, while others showed only a small benefit or none at all.…”

“…Neural speech tracking is widely used to study the neural processing of continuous speech, though primarily with audio-only stimuli (Brodbeck, Hong, et al, 2018;Chalas et al, 2022;Di Liberto et al, 2015;Keitel et al, 2018). Recent studies have used audiovisual speech settings, but without directly modeling the visual speech features Golumbic et al, 2013) or not incorporating their temporal dynamics due to the use of frequency-based methods (Aller et al, 2022;Bröhl et al, 2022;Park et al, 2016). Here, we show, for the first time, the temporal dynamics and cortical origins of TRFs obtained from lip movements in an audiovisual setting with one or two speakers.…”

Intelligibility improves perception of timing changes in speech

“…The number of channels also determines whether noise-vocoded stimuli are intelligible: 16-channel noise-vocoded speech is clearly intelligible, even for naïve listeners, whereas 1-channel noise-vocoded speech sounds like noise to naïve listeners and is completely unintelligible when presented in isolation. 1-channel vocoded speech yields <5% word report accuracy when presented alone, but ~30% when combined with visual cues like lip movements [ 22 ], suggesting that noise-vocoded speech can be perceived as speech-like, even when unintelligible. Importantly, however, these manipulations reveal changes in speech intelligibility that are independent of amplitude modulations, since these are very similar for 1-channel and 16-channel speech [ 23 ].…”

Visual speech cues enhance neural speech tracking in right auditory cluster leading to improvement in speech in noise comprehension in older adults with hearing impairment

“…Several studies suggest that older adults with high-frequency hearing loss in particular benefit from visual speech cues and show improved SiN performance (Altieri & Hudock, 2014;Hallam & Corney, 2014;Lidestam et al, 2014;Winneke & Phillips, 2011), while other studies clearly find a benefit to speech perception from audio-visual speech presentation, but independent of age and the degree of hearing loss and with considerable individual variability in the extent of this benefit (Başkent & Bazo, 2011;Rosemann & Thiel, 2018;Sommers et al, 2005;Tye-Murray et al, 2007). At the neural level, visual speech cues appear to enhance neural tracking of the speech envelope (Aller et al, 2022;Crosse et al, 2016;Micheli et al, 2018;Park et al, 2016) and restore early cortical tracking of speech presented in noise, complementing impaired auditory input (Atilgan et al, 2018;Crosse et al, 2015;Zion Golumbic et al, 2013). This audio-visual enhancement of neural speech tracking has also been observed in older individuals (Puschmann et al, 2019), although the relationship to SiN perception and comprehension has not yet been sufficiently established.…”