Neural correlates of cross-modal binding

Bushara, Khalafalla O.; Hanakawa, Takashi; Immisch, Ilka; Toma, Kazunori; Kansaku, Kenji; Hallett, Mark

doi:10.1038/nn993

Cited by 211 publications

(171 citation statements)

References 16 publications

Supporting

Mentioning

158

Contrasting

Order By: Relevance

“…2 shows the grand averaged responses obtained for each place-of-articulation condition tested in A, V, and AV conditions in experiment 1. The presence of visual speech (AV condition) significantly reduced the amplitude of the N1 and P2 auditory ERPs compared to auditory-alone conditions (A), in agreement with the deactivation hypothesis (31,32) and contrary to the expectation of supra-additivity.…”

Section: Methodssupporting

confidence: 72%

“…In fact, multisensory integration sites are found throughout the cortex (27), and a fundamental contribution of multisensory neurons may reside in the weighting of one sensory stream against the other, i.e., in reducing stimulus uncertainty (28), rather than in establishing a multisensory perceptual representation. Consistent with this view, complex patterns of activation have been reported that show suppression of sensory-specific cortices (29)(30)(31) in conjunction with enhanced activation of multisensory sites. In congruent AV speech, subadditive interactions in polysensory regions have also been observed (32).…”

Section: Neurophysiological Basis Of Multisensory Integrationmentioning

confidence: 60%

See 1 more Smart Citation

Visual speech speeds up the neural processing of auditory speech

Wassenhove

Grant

Poeppel

2005

Proc. Natl. Acad. Sci. U.S.A.

780

115

831

View full text Add to dashboard Cite

Synchronous presentation of stimuli to the auditory and visual systems can modify the formation of a percept in either modality. For example, perception of auditory speech is improved when the speaker's facial articulatory movements are visible. Neural convergence onto multisensory sites exhibiting supra-additivity has been proposed as the principal mechanism for integration. Recent findings, however, have suggested that putative sensory-specific cortices are responsive to inputs presented through a different modality. Consequently, when and where audiovisual representations emerge remain unsettled. In combined psychophysical and electroencephalography experiments we show that visual speech speeds up the cortical processing of auditory signals early (within 100 ms of signal onset). The auditory-visual interaction is reflected as an articulator-specific temporal facilitation (as well as a nonspecific amplitude reduction). The latency facilitation systematically depends on the degree to which the visual signal predicts possible auditory targets. The observed auditory-visual data support the view that there exist abstract internal representations that constrain the analysis of subsequent speech inputs. This is evidence for the existence of an ''analysis-by-synthesis'' mechanism in auditory-visual speech perception. combinations'' such as ''pk '' or ''kp '' but never a fused percept. These results illustrate the effect of input modality on the perceptual AV speech outcome and suggest that multisensory percept formation is systematically based on the informational content of the inputs. In classic speech theories, however, visual speech has seldom been accounted for as a natural source of speech input. Ultimately, when in the processing stream (i.e., at which representational stage) sensory-specific information fuses to yield unified percepts is fundamental for any theoretical, computational, and neuroscientific accounts of speech perception.Recent investigations of AV speech are based on hemodynamic studies that cannot speak directly to timing issues (2, 3). Electroencephalographic (EEG) and magnetoencephalographic (4-7) studies testing AV speech integration have typically used oddball or mismatch negativity paradigms, thus the earliest AV speech interactions have been reported for the 150-to 250-ms mismatch response. Whether systematic AV speech interactions can be documented earlier is controversial, although nonspeech effects can be observed early (8). AV Speech as a Multisensory ProblemSeveral properties of speech are relevant to the present study. (i) Because AV speech is ecologically valid for humans (9, 10), one might predict an involvement of specialized neural computations capable of handling the spectrotemporal complexity of AV speech (compared to, say, arbitrary tone-flash pairings), for which no natural functional relevance can be assumed. (ii) Natural AV speech is characterized by particular dynamics such as (a) the temporal precedence of visual speech (the movement of the facial articulators typically ...

show abstract

Section: Methodssupporting

confidence: 72%

Section: Neurophysiological Basis Of Multisensory Integrationmentioning

confidence: 60%

Visual speech speeds up the neural processing of auditory speech

Wassenhove

Grant

Poeppel

2005

Proc. Natl. Acad. Sci. U.S.A.

780

115

831

View full text Add to dashboard Cite

show abstract

“…Such a result would suggest that distraction is minimized by increasing attention to goal-relevant sensory representations, because orienting attention to a particular stimulus feature increases activity within cortical areas that process that feature (Shulman et al, 1999;Hopfinger et al, 2000;Giesbrecht et al, 2003;Woldorff et al, 2004). Because stimuli presented in different sensory modalities are processed in distinct brain regions (Bushara et al, 2003), the present study provided a particularly clear test of this hypothesis.…”

Section: Introductionmentioning

confidence: 74%

The Neural Mechanisms for Minimizing Cross-Modal Distraction

Weissman¹,

Warner²,

Woldorff³

2004

J. Neurosci.

145

134

View full text Add to dashboard Cite

The neural circuitry that increases attention to goal-relevant stimuli when we are in danger of becoming distracted is a matter of active debate. To address several long-standing controversies, we asked participants to identify a letter presented either visually or auditorily while we varied the amount of cross-modal distraction from an irrelevant letter in the opposite modality. Functional magnetic resonance imaging revealed three novel results. First, activity in sensory cortices that processed the relevant letter increased as the irrelevant letter became more distracting, consistent with a selective increase of attention to the relevant letter. In line with this view, an across-subjects correlation indicated that the larger the increase of activity in sensory cortices that processed the relevant letter, the less behavioral interference there was from the irrelevant letter. Second, regions of the dorsolateral prefrontal cortex (DLPFC) involved in orienting attention to the relevant letter also participated in increasing attention to the relevant letter when conflicting stimuli were present. Third, we observed a novel pattern of regional specialization within the cognitive division of the anterior cingulate cortex (ACC) for focusing attention on the relevant letter (dorsal ACC) versus detecting conflict from the irrelevant letter (rostral ACC). These findings indicate novel roles for sensory cortices, the DLPFC, and the ACC in increasing attention to goal-relevant stimulus representations when distracting stimuli conflict with behavioral objectives. Furthermore, they potentially resolve a long-standing controversy regarding the key contribution of the ACC to cognitive control.

show abstract

“…3). This system affords the possibility of potential future studies of cross-modal integration both behaviorally and neurophysiologically, topics that have received increased attention of late (5,(23)(24)(25)(26)(27)(28).…”

Section: Discussionmentioning

confidence: 99%

Cross-modal integration in a dart-poison frog

Narins

Grabul²,

Soma³

et al. 2005

Proc. Natl. Acad. Sci. U.S.A.

135

View full text Add to dashboard Cite

The mechanisms by which the brain binds together inputs from separate sensory modalities to effect a unified percept of events are poorly understood. This phenomenon was studied in males of the dart-poison frog Epipedobates femoralis. These animals physically and vigorously defend their territories against conspecific calling intruders. In prior field studies with an electromechanical model frog, we were able to experimentally evoke this aggressive behavior only when an auditory cue (advertisement call) was presented simultaneously with a visual cue (vocal-sac pulsations). In the present field experiments, we used a modified version of the electromechanical model frog to present territorial males with visual and auditory cues separated by experimentally introduced temporal delays or spatial disparities to probe temporal and spatial integration in this animal. In temporal integration experiments, bimodal stimuli with temporal overlap during calling bouts consistently evoked aggressive behavior; stimuli lacking bimodal temporal overlap were relatively ineffective at the same task. In spatial integration studies, despite presenting the components of the bimodal stimulus with an initial spatial disparity of up to 12 cm, fighting behavior persisted. These results demonstrate that temporal and spatial integration may be reliably estimated in a freely behaving animal in its natural habitat and that we can use aggressive behavior in this species as an index of cross-modal integration in the field.animal communication ͉ territorial defense ͉ anuran ͉ amphibian ͉ Dendrobatidae

show abstract

Neural correlates of cross-modal binding

Cited by 211 publications

References 16 publications

Visual speech speeds up the neural processing of auditory speech

Visual speech speeds up the neural processing of auditory speech

The Neural Mechanisms for Minimizing Cross-Modal Distraction

Cross-modal integration in a dart-poison frog

Contact Info

Product

Resources

About