Vision perceptually restores auditory spectral dynamics in speech

Abstract

<jats:p>Visual speech facilitates auditory speech perception, but the visual cues responsible for these benefits and the information they provide remain unclear. Low-level models emphasize basic temporal cues provided by mouth movements, but these impoverished signals may not fully account for the richness of auditory information provided by visual speech. High-level models posit interactions among abstract categorical (i.e., phonemes/visemes) or amodal (e.g., articulatory) speech representations, but require lossy remapping of speech signals onto abstracted representations. Because visible articulators shape the spectral content of speech, we hypothesized that the perceptual system might exploit natural correlations between midlevel visual (oral deformations) and auditory speech features (frequency modulations) to extract detailed spectrotemporal information from visual speech without employing high-level abstractions. Consistent with this hypothesis, we found that the time–frequency dynamics of oral resonances (formants) could be predicted with unexpectedly high precision from the changing shape of the mouth during speech. When isolated from other speech cues, speech-based shape deformations improved perceptual sensitivity for corresponding frequency modulations, suggesting that listeners could exploit this cross-modal correspondence to facilitate perception. To test whether this type of correspondence could improve speech comprehension, we selectively degraded the spectral or temporal dimensions of auditory sentence spectrograms to assess how well visual speech facilitated comprehension under each degradation condition. Visual speech produced drastically larger enhancements during spectral degradation, suggesting a condition-specific facilitation effect driven by cross-modal recovery of auditory speech spectra. The perceptual system may therefore use audiovisual correlations rooted in oral acoustics to extract detailed spectrotemporal information from visual speech.</jats:p>

Shared and modality-specific brain regions that mediate auditory and visual word comprehension

“…While acoustic speech consists of temporal and spectral modulations of sound pressure, visual speech consists of movements of the mouth, head, and hands. Movements of the mouth, lips and tongue in particular provide both redundant and complementary information to acoustic cues ( Hall et al, 2005 ; Peelle and Sommers, 2015 ; Plass et al, 2019 ; Summerfield, 1992 ), and can help to enhance speech intelligibility in noisy environments and in a second language ( Navarra and Soto-Faraco, 2007 ; Sumby and Pollack, 1954 ; Yi et al, 2013 ). While a plethora of studies have investigated the cerebral mechanisms underlying speech in general, we still have a limited understanding of the networks specifically mediating visual speech perception, that is lip reading ( Bernstein and Liebenthal, 2014 ; Capek et al, 2008 ; Crosse et al, 2015 ).…”

Cortical tracking of unheard formant modulations derived from silently presented lip movements and its decline with age

“…Here we can show that the brain is also able to perform a more fine-coarsed tracking than initially thought by especially processing the spectral fine-details that are modulated near the lips, another potentially learned association between lip-near auditory cues (i.e. merged F2 and F3 formants) and lip movements (Plass et al, 2020). Additionally, it is not only formants that are subject to visuo-phonological transformation, but also the fundamental frequency, as seen in our results.…”

“…Those two formants fluctuate around 2500 Hz and tend to merge into a single peak when pronouncing certain consonant-vowel combinations (Badin et al, 1990). The mentioned merging process is taking place in the front region of the oral cavity and can therefore also be seen by observing lip movements (Plass et al, 2020). The formants were extracted at a rate of 200 Hz for the sake of simplicity and then downsampled to 150 Hz.…”

“…Particularly relevant spectral fine details are formant peaks around 2500 Hz, which are indicated to be modulated in the front cavity (Badin et al, 1990). This corresponds to expansion and contraction of the lips (Plass et al, 2020), thus having a relationship with certain lip movements and could therefore be extracted for important phonological cues. Furthermore, not only resonant frequencies, but also the fundamental frequency (or pitch contour) plays an important role in speech understanding in noisy environments (Hopkins et .…”

“…Recent behavioural evidence describes that spectral fine details can also be extracted by observation of lip movements (Plass et al, 2020). This raises the interesting question whether this information is also represented at the level of the visual cortex, analogous to the envelope as shown previously (Hauswald et al, 2018).…”