Visual and auditory cortices represent acoustic speech-related information during silent lip reading

Abstract

Speech is an intrinsically multisensory signal and seeing the speaker's lips forms a cornerstone of communication in acoustically impoverished environments. Still, it remains unclear how the brain exploits visual speech for comprehension and previous work debated whether lip signals are mainly processed along the auditory pathways or whether the visual system directly implements speech-related processes. To probe this question, we systematically characterized dynamic representations of multiple acoustic and visual speech-derived features in source localized MEG recordings that were obtained while participants listened to speech or viewed silent speech. Using a mutual-information framework we provide a comprehensive assessment of how well temporal and occipital cortices reflect the physically presented signals and speech-related features that were physically absent but may still be critical for comprehension. Our results demonstrate that both cortices are capable of a functionally specific form of multisensory restoration: during lip reading both reflect unheard acoustic features, with occipital regions emphasizing spectral information and temporal regions emphasizing the speech envelope. Importantly, the degree of envelope restoration was predictive of lip reading performance. These findings suggest that when seeing the speaker's lips the brain engages both visual and auditory pathways to support comprehension by exploiting multisensory correspondences between lip movements and spectro-temporal acoustic cues.

Beyond Linear Neural Envelope Tracking: A Mutual Information Approach

“…Researchers can apply it to unrelated sets of variables as well, where one assesses the effect of a variable while controlling for a third. Conditional MI was recently applied in the field of envelope tracking (Bröhl et al, 2022). Note, however, that the Gaussian copula approach has an upper limit on the number of dimensions one can use for accurate estimations of the covariance matrices.…”

Auditory cortex encodes lipreading information through spatially distributed activity

“…This level of detail complements the temporal resolution offered by MEG, as used by Brohl et al, enabling a more comprehensive understanding of how visual and auditory speech information is integrated at different neural levels. Our controlled study design, focusing on specific parameters like stimulus length and voice onset time (VOT), offers distinct advantages over naturalistic designs such as those used by Brohl et al(54). By minimizing confounds due to natural correlations between auditory and visual speech elements, we can more accurately isolate and test the encoding of visual speech information in the auditory cortex.…”