Visual context due to speech-reading suppresses the auditory response to acoustic interruptions in speech

Abstract

Speech reading enhances auditory perception in noise. One means by which this perceptual facilitation comes about is through information from visual networks reinforcing the encoding of the congruent speech signal by ignoring interfering acoustic signals. We tested this hypothesis neurophysiologically by acquiring EEG while individuals listened to words with a fixed portion of each word replaced by white noise. Congruent (meaningful) or incongruent (reversed frames) mouth movements accompanied the words. Individuals judged whether they heard the words as continuous (illusion) or interrupted (illusion failure) through the noise. We hypothesized that congruent, as opposed to incongruent, mouth movements should further enhance illusory perception by suppressing the auditory cortex's response to interruption onsets and offsets. Indeed, we found that the N1 auditory evoked potential (AEP) to noise onsets and offsets was reduced when individuals experienced the illusion during congruent, but not incongruent, audiovisual streams. This N1 inhibitory effect was most prominent at noise offsets, suggesting that visual influences on auditory perception are instigated to a greater extent during noisy periods. These findings suggest that visual context due to speech-reading disengages (inhibits) neural processes associated with interfering sounds (e.g., noisy interruptions) during speech perception.

Tolerance for audiovisual asynchrony is enhanced by the spectrotemporal fidelity of the speaker’s mouth movements and speech

“…This suggests that speech-reading may help restore phonetic representations of the degraded CI speech. Consistent with this premise, other studies that used speech with missing segments reported that speech-reading enhances the perception of continuity and intelligibility of the degraded speech (Benard & Başkent, 2015; Bhat, Pitt, & Shahin, 2014; Grant et al, 1998; Shahin, Bishop, & Miller, 2009; Shahin, Kerlin, Bhat, & Miller, 2012). …”

Quantifying lip‐read‐induced suppression and facilitation of the auditory N1 and P2 reveals peak enhancements and delays

“…As mentioned, the most widely used stimuli consist of phonetically matching and naturally timed AV speech. Therefore, data obtained with AV phonetic incongruent material (e.g., Alsius, Möttönen, Sams, Soto‐Faraco, & Tiippana, ), AV asynchronous stimuli (e.g., Huhn et al, ), or stimuli with artificial unimodal components (e.g., Baart, Stekelenburg, & Vroomen, ; Bhat, Pitt, & Shahin, ; Meyer, Harrison, & Wuerger, ) were excluded. Any work that did not include GAs for auditory speech as well as for AV(−V), or did not allow those conditions to be estimated, was excluded as well (Knowland, Mercure, Karmiloff‐Smith, Dick, & Thomas, ; Liu, Lin, Gao, & Dang, ; Magnée, de Gelder, van Engeland, & Kemner, ; Winkler, Horvath, Weisz, & Trejo, ), because study‐specific parameters that have an overall effect on the GAs can only be factored out when considering both A and AV(−V).…”

Neurophysiology underlying influence of stimulus reliability on audiovisual integration

“…For example, older adults, who are known to have lower acoustic temporal resolution, show smaller N1-P2 AEPs compared to younger adults for the same gaps (Michalewski et al, 2005;Harris et al, 2012). Also, visually driven suppression of N1-P2 AEPs to onsets and offsets of gaps in speech reduces the temporal resolution to gaps and strengthens the perception of auditory continuity (Bhat et al, 2014).…”