Effect of attentional load on audiovisual speech perception: evidence from ERPs

Abstract

Seeing articulatory movements influences perception of auditory speech. This is often reflected in a shortened latency of auditory event-related potentials (ERPs) generated in the auditory cortex. The present study addressed whether this early neural correlate of audiovisual interaction is modulated by attention. We recorded ERPs in 15 subjects while they were presented with auditory, visual, and audiovisual spoken syllables. Audiovisual stimuli consisted of incongruent auditory and visual components known to elicit a McGurk effect, i.e., a visually driven alteration in the auditory speech percept. In a Dual task condition, participants were asked to identify spoken syllables whilst monitoring a rapid visual stream of pictures for targets, i.e., they had to divide their attention. In a Single task condition, participants identified the syllables without any other tasks, i.e., they were asked to ignore the pictures and focus their attention fully on the spoken syllables. The McGurk effect was weaker in the Dual task than in the Single task condition, indicating an effect of attentional load on audiovisual speech perception. Early auditory ERP components, N1 and P2, peaked earlier to audiovisual stimuli than to auditory stimuli when attention was fully focused on syllables, indicating neurophysiological audiovisual interaction. This latency decrement was reduced when attention was loaded, suggesting that attention influences early neural processing of audiovisual speech. We conclude that reduced attention weakens the interaction between vision and audition in speech.

Physiology, Psychoacoustics and Cognition in Normal and Impaired Hearing

“…The results are consistent with the previous findings (Nahorna et al 2012) Navarra et al 2010;Alsius et al 2014). Experiment B shows that attentional mechanisms may intervene at the level of single audiovisual sources in an audiovisual speech scene, selectively increasing or decreasing the amount of fusion depending on the coherence of the attended source.…”

Turning a blind eye to the lexicon: ERPs show no cross-talk between lip-read and lexical context during speech sound processing

“…As demonstrated by McGurk and MacDonald (1976), lip-read context can change perceived sound identity, and when it does, it triggers an auditory MMN response when the illusory AV stimulus is embedded in a string of congruent AV stimuli (e.g., Colin, Radeau, Soquet, & Deltenre, 2004;Colin et al, 2002;Saint-Amour, De Sanctis, Molholm, Ritter, & Foxe, 2007). When sound onset is sudden and does not follow repeated presentations of standard sounds, it triggers an N1/P2 complex (a negative peak at 100 ms followed by a positive peak at $200 ms) and it is well-documented that amplitude and latency of both peaks are modulated by lip-read speech (e.g., Alsius, Möttönen, Sams, Soto-Faraco, & Tiippana, 2014;Baart, Stekelenburg, & Vroomen, 2014;Besle, Fort, Delpuech, & Giard, 2004;Frtusova, Winneke, & Phillips, 2013;Klucharev, Möttönen, & Sams, 2003;Stekelenburg, Maes, van Gool, Sitskoorn, & Vroomen, 2013;van Wassenhove, Grant, & Poeppel, 2005;Winneke & Phillips, 2011). Thus, studies measuring both the MMN and the N1/P2 peaks indicate that lip-reading affects sound processing within 200 to 250 ms after sound onset.…”

The COGs (context, object, and goals) in multisensory processing

“…van Wassenhove et al 2005) were found to be substantially reduced (or even eliminated; Alsius et al 2014). …”