Neural restoration of degraded audiovisual speech

Abstract

When speech is interrupted by noise, listeners often perceptually “fill-in” the degraded signal, giving an illusion of continuity and improving intelligibility. This phenomenon involves a neural process in which the auditory cortex (AC) response to onsets and offsets of acoustic interruptions is suppressed. Since meaningful visual cues behaviorally enhance this illusory filling-in, we hypothesized that during the illusion, lip movements congruent with acoustic speech should elicit a weaker AC response to interruptions relative to static (no movements) or incongruent visual speech. AC response to interruptions was measured as the power and inter-trial phase consistency of the auditory evoked theta band (4-8 Hz) activity of the electroencephalogram (EEG) and the N1 and P2 auditory evoked potentials (AEPs). A reduction in the N1 and P2 amplitudes and in theta phase-consistency reflected the perceptual illusion at the onset and/or offset of interruptions regardless of visual condition. These results suggest that the brain engages filling-in mechanisms throughout the interruption, which repairs degraded speech lasting up to ~250 ms following the onset of the degradation. Behaviorally, participants perceived greater speech continuity over longer interruptions for congruent compared to incongruent or static audiovisual streams. However, this specific behavioral profile was not mirrored in the neural markers of interest. We conclude that lip-reading enhances illusory perception of degraded speech not by altering the quality of the AC response, but by delaying it during degradations so that longer interruptions can be tolerated.

Hearing an Illusory Vowel in Noise: Suppression of Auditory Cortical Activity

“…Our finding that off-responses in right AC were partially suppressed when listeners experienced continuity illusions of the vowel (as opposed to veridical discontinuity percepts) is consistent with previous results on continuity illusions (Petkov et al, 2007;Riecke et al, 2009b;Shahin et al, 2012). Importantly, our findings extend the no-discontinuity rule for auditory restoration to vowels and reveal the underlying substrate (right AC) and interval (40 -280 ms after vowel offset).…”

“…We focused on the putative auditory cortical processes indexed by the lower-frequency power changes and investigated them under the acoustic/perceptual conditions depicted in Figure 4 (top). We hypothesized that neural activity specifically in the 4 Hz band is critical for vowel restoration because EEG power changes in this band have been associated with auditory restoration (Riecke et al, 2009b;Shahin et al, 2012), sound onsets (Makeig, 1993), and auditory change detection (Cacace and McFarland, 2003). These changes could resemble the major spectral component of auditory-evoked on/off responses, which also have been associated with tone restoration (Petkov et al, 2007).…”

“…Vocal pitch may improve the intelligibility of fragmented speech (Başkent and Chatterjee, 2010); thus, restorations of fragmented speech or other complex sounds (e.g., melodies) may also benefit from suppression of offsetevoked activity (Shahin et al, 2012). In this view, the extended no-discontinuity rule may further include that there exists no neural evidence of the offset of the complex sound features that belong to the target sound (Chait et al, 2008;ShinnCunningham and Wang, 2008).…”

“…This putative mechanism could also be involved in the restoration of ecologically valid sounds: the no-discontinuity rule has been validated behaviorally for speech (Başkent et al, 2009), and continuity illusions of vowels and speech have been associated with changes in hemodynamic activity in temporal and parietal cortices (Shahin et al, 2009;Riecke et al, 2011a), although neural on/off responses could not be resolved in these studies. Therefore, evidence for a role of onset/ offset suppression for the restoration of ecologically valid sounds is sparse (Shahin et al, 2012).…”

Dynamic cortical representations of perceptual filling-in for missing acoustic rhythm

“…Samuel's paradigm 22 ). Our method subscribes to the latter approach, also referred to as 'filling-in', which emphasizes the signal detection strategy followed in cases where a listener classification is inconsistent with the token absence in a gap 3,[23][24][25][26][27][28][29] . As has been noted 30 , from the listener's utilitarian perspective, this effect of induction in a challenging environment is not aimed at the production of decision errors (or illusions) but to assist against masking.…”