Capture of visual attention interferes with multisensory speech processing

Abstract

Attending to a conversation in a crowded scene requires selection of relevant information, while ignoring other distracting sensory input, such as speech signals from surrounding people. The neural mechanisms of how distracting stimuli influence the processing of attended speech are not well understood. In this high-density electroencephalography (EEG) study, we investigated how different types of speech and non-speech stimuli influence the processing of attended audiovisual speech. Participants were presented with three horizontally aligned speakers who produced syllables. The faces of the three speakers flickered at specific frequencies (19 Hz for flanking speakers and 25 Hz for the center speaker), which induced steady-state visual evoked potentials (SSVEP) in the EEG that served as a measure of visual attention. The participants' task was to detect an occasional audiovisual target syllable produced by the center speaker, while ignoring distracting signals originating from the two flanking speakers. In all experimental conditions the center speaker produced a bimodal audiovisual syllable. In three distraction conditions, which were contrasted with a no-distraction control condition, the flanking speakers either produced audiovisual speech, moved their lips, and produced acoustic noise, or moved their lips without producing an auditory signal. We observed behavioral interference in the reaction times (RTs) in particular when the flanking speakers produced naturalistic audiovisual speech. These effects were paralleled by enhanced 19 Hz SSVEP, indicative of a stimulus-driven capture of attention toward the interfering speakers. Our study provides evidence that non-relevant audiovisual speech signals serve as highly salient distractors, which capture attention in a stimulus-driven fashion.

Electrophysiological evidence of different neural processing between visual and audiovisual inhibition of return

“…They also found that the simultaneous occurrence of visual and auditory targets was integrated (i.e., multisensory integration, MSI). The integrated bimodal stimuli can help improve signal salience 8 and attract more attention 9 , 10 . The IOR actually reflect that attention is blocked from returning a previously attended location 11 and facilitates visual exploration by reducing the perceptual salience of a previously explored location 12 .…”

Audio-visual synchrony and spatial attention enhance processing of dynamic visual stimulation independently and in parallel: a frequency-tagging study

“…As a consequence, they would gain an automatic processing advantage over unisensory stimuli. In a task that requires a sustained focus of attention on a specific position in the visual field multisensory stimuli may then act as strong distractors (Krause et al, 2012) because they withdraw common processing resources from the task-relevant focus of attention.…”

Multisensory perception and action: development, decision-making, and neural mechanisms

“…Stekelenburg and Vroomen ( 2012 ) also showed that spatial congruity between auditory and visual signals modulates audiovisual interactions reflected in early ERP components, namely, the N1 and P2. Early integration may boost the saliency of the multisensory signals, even when the multisensory signals are irrelevant distractors, causing an attentional shift toward the multisensory distractor, as measured by steady-state visual evoked potentials (SSVEP) in an audiovisual speech task (Krause et al, 2012 ). Instead of using multisensory signals, Töllner et al ( 2012 ) presented separate auditory and visual signals in a dual-task paradigm requiring both auditory and visual discriminations, to investigate influences of task order predictability (TOP) and inter-task onset asynchrony (SOA) on perceptual, and motor processing stages, two stages indexed, respectively, by two EEG components: the Posterior-Contralateral- Negativity (PCN) and the Lateralized-Readiness-Potential (LRP).…”