A linguistic representation in the visual system underlies successful lipreading

Nidiffer, Aaron; Cao, Cody Zhewei; O’Sullivan, Aisling E.; Lalor, Edmund C.

doi:10.1101/2021.02.09.430299

Cited by 14 publications

(25 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This suggests a direct and comprehension-relevant link between the dynamics of the lip contour and spectral speech features (Campbell, 2008). Hence, a representation of acoustic features during silent lip reading may underlie the mapping of lip movements onto phonological units such as visemes, a form of language-specific representation emerging along visual pathways (Nidiffer et al, 2021; O’Sullivan et al, 2017).…”

Section: Discussionmentioning

confidence: 99%

Visual and auditory cortices represent acoustic speech-related information during silent lip reading

Bröhl

Keitel

Kayser

2022

Preprint

View full text Add to dashboard Cite

Speech is an intrinsically multisensory signal and seeing the speaker's lips forms a cornerstone of communication in acoustically impoverished environments. Still, it remains unclear how the brain exploits visual speech for comprehension and previous work debated whether lip signals are mainly processed along the auditory pathways or whether the visual system directly implements speech-related processes. To probe this question, we systematically characterized dynamic representations of multiple acoustic and visual speech-derived features in source localized MEG recordings that were obtained while participants listened to speech or viewed silent speech. Using a mutual-information framework we provide a comprehensive assessment of how well temporal and occipital cortices reflect the physically presented signals and speech-related features that were physically absent but may still be critical for comprehension. Our results demonstrate that both cortices are capable of a functionally specific form of multisensory restoration: during lip reading both reflect unheard acoustic features, with occipital regions emphasizing spectral information and temporal regions emphasizing the speech envelope. Importantly, the degree of envelope restoration was predictive of lip reading performance. These findings suggest that when seeing the speaker's lips the brain engages both visual and auditory pathways to support comprehension by exploiting multisensory correspondences between lip movements and spectro-temporal acoustic cues.

show abstract

Section: Discussionmentioning

confidence: 99%

Visual and auditory cortices represent acoustic speech-related information during silent lip reading

Bröhl

Keitel

Kayser

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A recent study using electrocorticography similarly demonstrated that medial occipital cortex exhibit reliable auditory envelope tracking in the absence of visual speech (Micheli et al, 2020). Other studies have suggested that visual cortex represents the unheard auditory speech during silent lipreading in the form of its amplitude envelope (Hauswald et al, 2018) and higher-level linguistic feature representations (Nidiffer et al, 2021; Suess et al, 2021). Correspondingly, we also found evidence of visual cortex tracking the unheard auditory speech envelope in silent lipreading (Fig.…”

Section: Discussionmentioning

confidence: 99%

“…Auditory prediction errors (and hence auditory phase-locking) arise in visual cortical areas when degraded speech sounds cannot accurately predict visual speech cues during audio-visual speech perception. The resulting prediction errors signal viseme information (Nidiffer et al, 2021) that can be used to update higher-level interpretations and support optimal speech perception when visual and auditory stimuli must be combined (Olasagasti et al, 2015).…”

Section: Cross-modal Prediction Of Audio and Visual Speech Signalsmentioning

confidence: 99%

“…The prediction follows, that neural responses also track visual speech signals. Indeed, neural responses entrain to speakers’ lip movements in various listening situations: clear audio-visual speech from a single speaker (Luo et al, 2010; Micheli et al, 2020; Mégevand et al, 2020), clear audio-visual speech from multiple speakers (Zion Golumbic et al, 2013; Park et al, 2016), silent visual-only speech (O’Sullivan et al, 2017; Hauswald et al, 2018; Bourguignon et al, 2020; Nidiffer et al, 2021), and audio-visual speech-in-noise (Keitel et al, 2018). Here, we investigated how visual and auditory speech signals are combined to support comprehension without segregation of speech and background noise.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Differential auditory and visual phase-locking are observed during audio-visual benefit and silent lip-reading for speech perception

Aller

Økland

MacGregor

et al. 2021

Preprint

View full text Add to dashboard Cite

Speech perception in noisy environments is enhanced by seeing facial movements of communication partners. However, the neural mechanisms by which audio and visual speech are combined are not fully understood. We explore MEG phase locking to auditory and visual signals in MEG recordings from 14 human participants (6 female) that reported words from single spoken sentences. We manipulated the acoustic clarity and visual speech signals such that critical speech information is present in auditory, visual or both modalities. MEG coherence analysis revealed that both auditory and visual speech envelopes (auditory amplitude modulations and lip aperture changes) were phase-locked to 2-6Hz brain responses in auditory and visual cortex, consistent with entrainment to syllable-rate components. Partial coherence analysis was used to separate neural responses to correlated audio-visual signals and showed non-zero phase locking to auditory envelope in occipital cortex during audio-visual (AV) speech. Furthermore, phase-locking to auditory signals in visual cortex was enhanced for AV speech compared to audio-only (AO) speech that was matched for intelligibility. Conversely, auditory regions of the superior temporal gyrus (STG) did not show above-chance partial coherence with visual speech signals during AV conditions, but did show partial coherence in VO conditions. Hence, visual speech enabled stronger phase locking to auditory signals in visual areas, whereas phase-locking of visual speech in auditory regions only occurred during silent lip-reading. Differences in these cross-modal interactions between auditory and visual speech signals are interpreted in line with cross-modal predictive mechanisms during speech perception.

show abstract

“…They provided evidence for a linguistic representation in the visual cortex stemming from visemic information of speech, a process independent of auditory processing associated with lip reading (Nidiffer et al, 2021).…”

Section: Occlusion Of Lip Movements Impairs Tracking Of Higher-level Segmentational Features Especially In Challenging Listening Situatiomentioning

confidence: 99%

Occlusion of lip movements impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

Haider

Suess

Hauswald

et al. 2021

Preprint

View full text Add to dashboard Cite

Face masks have become a prevalent measure during the Covid-19 pandemic to counteract the transmission of SARS-CoV 2. An unintended 'side-effect' of face masks is their adverse influence on speech perception especially in challenging listening situations. So far, behavioural studies have not pinpointed exactly which feature(s) of speech processing face masks affect in such listening situations. We conducted an audiovisual (AV) multi-speaker experiment using naturalistic speech (i.e. an audiobook). In half of the trials, the target speaker wore a (surgical) face mask, while we measured the brain activity of normal hearing participants via magnetoencephalography (MEG). A decoding model on the clear AV speech (i.e. no additional speaker and target speaker not wearing a face mask) was trained and used to reconstruct crucial speech features in each condition. We found significant main effects of face masks on the reconstruction of acoustic features, such as the speech envelope and spectral speech features (i.e. pitch and formant frequencies), while reconstruction of higher level features of speech segmentation (phoneme and word onsets) were especially impaired through masks in difficult listening situations, i.e. when a distracting speaker was also presented. Our findings demonstrate the detrimental impact face masks have on listening and speech perception, thus extending previous behavioural results. Supporting the idea of visual facilitation of speech is the fact that we used surgical face masks in our study, which only show mild effects on speech acoustics. This idea is in line with recent research, also by our group, showing that visual cortical regions track spectral modulations. Since hearing impairment usually affects higher frequencies, the detrimental effect of face masks might pose a particular challenge for individuals who likely need the visual information about higher frequencies (e.g. formants) to compensate.

show abstract

A linguistic representation in the visual system underlies successful lipreading

Cited by 14 publications

References 79 publications

Visual and auditory cortices represent acoustic speech-related information during silent lip reading

Visual and auditory cortices represent acoustic speech-related information during silent lip reading

Differential auditory and visual phase-locking are observed during audio-visual benefit and silent lip-reading for speech perception

Occlusion of lip movements impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

Contact Info

Product

Resources

About