Spatio‐temporal distribution of brain activity associated with audio‐visually congruent and incongruent speech and the McGurk Effect

Abstract

IntroductionSpatio‐temporal distributions of cortical activity to audio‐visual presentations of meaningless vowel‐consonant‐vowels and the effects of audio‐visual congruence/incongruence, with emphasis on the McGurk effect, were studied. The McGurk effect occurs when a clearly audible syllable with one consonant, is presented simultaneously with a visual presentation of a face articulating a syllable with a different consonant and the resulting percept is a syllable with a consonant other than the auditorily presented one.MethodsTwenty subjects listened to pairs of audio‐visually congruent or incongruent utterances and indicated whether pair members were the same or not. Source current densities of event‐related potentials to the first utterance in the pair were estimated and effects of stimulus–response combinations, brain area, hemisphere, and clarity of visual articulation were assessed.ResultsAuditory cortex, superior parietal cortex, and middle temporal cortex were the most consistently involved areas across experimental conditions. Early (<200 msec) processing of the consonant was overall prominent in the left hemisphere, except right hemisphere prominence in superior parietal cortex and secondary visual cortex. Clarity of visual articulation impacted activity in secondary visual cortex and Wernicke's area. McGurk perception was associated with decreased activity in primary and secondary auditory cortices and Wernicke's area before 100 msec, increased activity around 100 msec which decreased again around 180 msec. Activity in Broca's area was unaffected by McGurk perception and was only increased to congruent audio‐visual stimuli 30–70 msec following consonant onset.ConclusionsThe results suggest left hemisphere prominence in the effects of stimulus and response conditions on eight brain areas involved in dynamically distributed parallel processing of audio‐visual integration. Initially (30–70 msec) subcortical contributions to auditory cortex, superior parietal cortex, and middle temporal cortex occur. During 100–140 msec, peristriate visual influences and Wernicke's area join in the processing. Resolution of incongruent audio‐visual inputs is then attempted, and if successful, McGurk perception occurs and cortical activity in left hemisphere further increases between 170 and 260 msec.

Audiovisual incongruence differentially impacts left and right hemisphere sensorimotor oscillations: Potential applications to production

“…It may then be proposed that in the McGurk paradigm, both visual and auditory streams give rise to independent articulatory hypotheses, which must be reconciled to allow the extraction of a unified phonological representation during working memory encoding. This assertion is consistent with previous investigations of the McGurk paradigm which have proposed an initial discrepancy detection stage followed by later resolution/integration processes [ 48 – 50 ]. In congruent (i.e., audiovisual match) trials, minimal discrepancy is anticipated between initial visual-based and auditory-based articulatory hypotheses, allowing for rapid phonological encoding in working memory.…”

“…In the current study, ~15% of subjects (6/39) were classified as non-perceivers and excluded from further analysis as they did not perceive sensory fusion in at least 15% of McGurk trials [ 49 ]. The proportion of subjects not susceptible to the McGurk illusion is not universally reported in the literature, nor is there a standardized threshold subjects must meet to be considered a ‘perceiver / non-perceiver.’ This consideration notwithstanding, reported ranges for the proportion of non-perceiving subjects range from 0–54% [ 48 , 79 ] with an average of ~25% [ 54 ]. Among the subjects in the current study who reliably perceived the McGurk illusion, sensory fusion was reported in 73.5% of trials.…”

“…Among the subjects in the current study who reliably perceived the McGurk illusion, sensory fusion was reported in 73.5% of trials. While there is considerable variability across studies, published averages for the proportion of trials inducing the McGurk illusion range from 32–84% [ 48 , 49 , 80 ]. Given the reliable elicitation of the McGurk illusion in the current study, as well as the concordance with previous reports regarding both the proportion of perceiving subjects and trials eliciting sensory fusion, neural data are interpreted through the lens of the McGurk effect.…”

Neural Correlates of Modality-Sensitive Deviance Detection in the Audiovisual Oddball Paradigm

“…These included channel groups corresponding to established sites of auditory, audiovisual, and visual speech processing, including frontal, left temporal, and central parietal channels [ 23 , 24 , 31 , 46 , 47 , 48 , 49 , 50 , 51 ]. Consistent with previous audiovisual oddball studies, adaptive means and peak latencies were calculated in 4 time windows of interest corresponding to the N1 (50–100 ms), P2 (100–150 ms), early MMN (150–200 ms), and late MMN (300–400 ms) [ 2 , 24 , 25 , 52 ]. Peak latencies were defined as the time of maximum voltage in time windows of interest, and adaptive means were calculated for each participant by averaging 10 samples (20 ms) on either side of the peak.…”

Comparison of Resting-State Spontaneous Brain Activity between Treatment-Naive Patients with Schizophrenia and Obsessive-Compulsive Disorder

“…Among these present findings, the abnormalities of ALFF, ReHo and DC in the angular gyrus were especially noticeable. Recent researches have suggested that the angular gyrus is responsible for complex mental phenomena and processes, such as understanding visual and audio inputs [81], interpreting languages [82], retrieving memories [83], and maintaining consciousness [84]. Moreover, the angular gyrus has been demonstrated to be one of the overlapping regions between the default mode network and social brain networks [85].…”