Neural correlates of audiovisual speech processing in a second language

Barrós‐Loscertales, Alfonso; Ventura‐Campos, Noelia; Visser, Maya; Alsius, Agnès; Pallier, Christophe; Rivera, César Ávila; Soto‐Faraco, Salvador

doi:10.1016/j.bandl.2013.05.009

Cited by 16 publications

(11 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From a functional perspective, our results are in keeping with the notion that during speech perception, the auditory and visual sensory systems interact at multiple levels of processing (Schwartz et al, 1998; Nahorna et al, 2012; Barrós-Loscertales et al, 2013), and that top-down modulatory signals can influence at least some of these levels. Multisensory links do not solely rely on feed-forward convergence from unisensory regions to multisensory brain areas, but also implicate back-projections from association areas to multiple levels of (early) sensory processing that are based on current task demands (Calvert et al, 1999, 2000; Macaluso et al, 2000; Friston, 2005; Driver and Noesselt, 2008).…”

Section: Discussionsupporting

confidence: 89%

Effect of attentional load on audiovisual speech perception: evidence from ERPs

et al. 2014

Self Cite

View full text Add to dashboard Cite

Seeing articulatory movements influences perception of auditory speech. This is often reflected in a shortened latency of auditory event-related potentials (ERPs) generated in the auditory cortex. The present study addressed whether this early neural correlate of audiovisual interaction is modulated by attention. We recorded ERPs in 15 subjects while they were presented with auditory, visual, and audiovisual spoken syllables. Audiovisual stimuli consisted of incongruent auditory and visual components known to elicit a McGurk effect, i.e., a visually driven alteration in the auditory speech percept. In a Dual task condition, participants were asked to identify spoken syllables whilst monitoring a rapid visual stream of pictures for targets, i.e., they had to divide their attention. In a Single task condition, participants identified the syllables without any other tasks, i.e., they were asked to ignore the pictures and focus their attention fully on the spoken syllables. The McGurk effect was weaker in the Dual task than in the Single task condition, indicating an effect of attentional load on audiovisual speech perception. Early auditory ERP components, N1 and P2, peaked earlier to audiovisual stimuli than to auditory stimuli when attention was fully focused on syllables, indicating neurophysiological audiovisual interaction. This latency decrement was reduced when attention was loaded, suggesting that attention influences early neural processing of audiovisual speech. We conclude that reduced attention weakens the interaction between vision and audition in speech.

show abstract

Section: Discussionsupporting

confidence: 89%

Effect of attentional load on audiovisual speech perception: evidence from ERPs

et al. 2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…In a recent example, Man et al (2012) demonstrated similar neural activity patterns in the left pSTS for non-speech visual-only representation and acoustic-only representation of the same object. Supporting our findings, the left pSTS has been consistently recruited in AV language studies using the max criterion for AV integration (conjunction of AV > A O and AV > V O ; Beauchamp, 2005b) of congruent AV stimuli including various stimulus types, such as sentences in native and non-native language (Barros-Loscertales et al, 2013), words (Szycik et al, 2008), and visual letters paired with speech sounds (van Atteveldt et al, 2004, 2007). Similarly, the left pSTS showed increased activity to congruent AV story stimuli compared to the sum of activity for acoustic-only and visual-only stimulation (Calvert et al, 2000); others have also reported supra-additive AV speech effects in STS (Wright et al, 2003).…”

Section: Discussionsupporting

confidence: 83%

“…We suggest that this method allowed for the isolation of AV-processing regions most likely to be involved in processing congruent AV speech or the change in perception accompanying the McGurk effect. This statistical approach has been successfully utilized to isolate AV-processing regions in several language studies (van Atteveldt et al, 2004, 2007; Szycik et al, 2008; Barros-Loscertales et al, 2013) and other types of AV studies (Beauchamp, 2005b; Hein et al, 2007; Watson et al, 2014). Since others have raised the issue of high individual anatomical/functional variability concerning the multisensory portion of the STS (Beauchamp et al, 2010; Nath and Beauchamp, 2012), we confirmed our group results in single-subject analyses, accounting for individual differences in gyral anatomy (Geschwind and Levitsky, 1968) and functional localization within pST.…”

Section: Introductionmentioning

confidence: 99%

Distinct cortical locations for integration of audiovisual speech and the McGurk effect

Erickson

Zielinski

et al. 2014

Front. Psychol.

View full text Add to dashboard Cite

Audiovisual (AV) speech integration is often studied using the McGurk effect, where the combination of specific incongruent auditory and visual speech cues produces the perception of a third illusory speech percept. Recently, several studies have implicated the posterior superior temporal sulcus (pSTS) in the McGurk effect; however, the exact roles of the pSTS and other brain areas in “correcting” differing AV sensory inputs remain unclear. Using functional magnetic resonance imaging (fMRI) in ten participants, we aimed to isolate brain areas specifically involved in processing congruent AV speech and the McGurk effect. Speech stimuli were composed of sounds and/or videos of consonant–vowel tokens resulting in four stimulus classes: congruent AV speech (AVCong), incongruent AV speech resulting in the McGurk effect (AVMcGurk), acoustic-only speech (AO), and visual-only speech (VO). In group- and single-subject analyses, left pSTS exhibited significantly greater fMRI signal for congruent AV speech (i.e., AVCong trials) than for both AO and VO trials. Right superior temporal gyrus, medial prefrontal cortex, and cerebellum were also identified. For McGurk speech (i.e., AVMcGurk trials), two clusters in the left posterior superior temporal gyrus (pSTG), just posterior to Heschl’s gyrus or on its border, exhibited greater fMRI signal than both AO and VO trials. We propose that while some brain areas, such as left pSTS, may be more critical for the integration of AV speech, other areas, such as left pSTG, may generate the “corrected” or merged percept arising from conflicting auditory and visual cues (i.e., as in the McGurk effect). These findings are consistent with the concept that posterior superior temporal areas represent part of a “dorsal auditory stream,” which is involved in multisensory integration, sensorimotor control, and optimal state estimation (Rauschecker and Scott, 2009).

show abstract

“…Others have proposed that many more sensory areas than previously assumed may have multimodal properties (Driver and Noesselt, 2008; Ghazanfar and Schroeder, 2006; Hackett and Schroeder, 2009), and previous studies have shown plasticity of sensory areas in blind or deaf individuals (Amedi et al, 2003; Amedi et al, 2007; Bavelier and Neville, 2002; Bedny et al, 2011; Finney et al, 2001; Rauschecker, 1995; Renier et al, 2010; Striem-Amit and Amedi, 2014; Striem-Amit et al, 2012; Weeks et al, 2000). A recent study of non-native, second language processing recruited bilateral occipital cortex during congruent versus incongruent stimulation of AV sentences (Barros-Loscertales et al, 2013). Other studies have shown FFG activation in voice/speaker recognition tasks of auditory-only speech (von Kriegstein et al, 2005), and FFG recruitment during face processing (Haxby et al, 2000; Hoffman and Haxby, 2000).…”

Section: Discussionmentioning

confidence: 99%

An ALE meta‐analysis on the audiovisual integration of speech signals

Erickson

Heeg

Rauschecker

et al. 2014

Human Brain Mapping

View full text Add to dashboard Cite

The brain improves speech processing through the integration of audiovisual (AV) signals. Situations involving AV speech integration may be crudely dichotomized into those where auditory and visual inputs contain 1) equivalent, complementary signals (validating AV speech), or 2) inconsistent, different signals (conflicting AV speech). This simple framework may allow for the systematic examination of broad commonalities and differences between AV neural processes engaged by various experimental paradigms frequently used to study AV speech integration. We conducted an activation likelihood estimation (ALE) meta-analysis of 22 functional imaging studies comprising 33 experiments, 311 subjects, and 347 foci examining “conflicting” versus “validating” AV speech. Experimental paradigms included content congruency, timing synchrony, and perceptual measures, such as the McGurk effect or synchrony judgments, across AV speech stimulus types (sub-lexical to sentence). Co-localization of conflicting AV speech experiments revealed consistency across at least two contrast types (e.g., synchrony and congruency) in a network of dorsal-stream regions in the frontal, parietal, and temporal lobes. There was consistency across all contrast types (synchrony, congruency, and percept) in the bilateral posterior superior/middle temporal cortex. Although fewer studies were available, validating AV speech experiments were localized to other regions, such as ventral-stream visual areas in the occipital and inferior temporal cortex. These results suggest that while equivalent, complementary AV speech signals may evoke activity in regions related to the corroboration of sensory input, conflicting AV speech signals recruit widespread dorsal-stream areas likely involved in the resolution of conflicting sensory signals.

show abstract

Neural correlates of audiovisual speech processing in a second language

Cited by 16 publications

References 54 publications

Effect of attentional load on audiovisual speech perception: evidence from ERPs

Effect of attentional load on audiovisual speech perception: evidence from ERPs

Distinct cortical locations for integration of audiovisual speech and the McGurk effect

An ALE meta‐analysis on the audiovisual integration of speech signals

Contact Info

Product

Resources

About