Neural Correlates of Auditory–Visual Stimulus Onset Asynchrony Detection

Abstract

Intersensory temporal synchrony is an ubiquitous sensory attribute that has proven to be critical for binding multisensory inputs, sometimes erroneously leading to dramatic perceptual illusions. However, little is known about how the brain detects temporal synchrony between multimodal sensory inputs. We used positron emission tomography to demonstrate that detecting auditory-visual stimulus onset asynchrony activates a large-scale neural network of insular, posterior parietal, prefrontal, and cerebellar areas with the highest and task-specific activity localized to the right insula. Interregional covariance analysis further showed significant task-related functional interactions between the insula, the posterior thalamus, and superior colliculus. Based on these results and the available electrophysiological and anatomical connectivity data in animals, we propose that the insula, via its known short-latency connections with the tectal system, mediates temporally defined auditory-visual interaction at an early stage of cortical processing permitting phenomena such as the ventriloquist and the McGurk illusions.Key words: audiovisual asynchrony; temporal integration; insular cortex; PET; multisensory; cortical processing A fundamental brain function is to integrate information available to multiple sensory modalities producing a unified representation of the external world. Although multimodal sensory inputs from a single object or event normally coincide both in space and time, intersensory integration mechanisms seem to rely more critically on their temporal than spatial congruence (e.g., the ventriloquist effect) (Bertelson and Radeau, 1981). Indeed, the ability to detect and use temporal synchrony in associating multimodal sensory stimuli (e.g., sounds and visual events) has been demonstrated in human infants as young as 2 months (Lewkowicz, 1996(Lewkowicz, , 2000, and is believed to be operational at birth providing an innate capacity on which intermodal perception and associative learning are based (Spelke, 1987;Bahrick, 1992). Yet, the neural correlates for this basic process remain unknown. Using positron emission tomography (PET), we studied normal human subjects while performing a task requiring detection of auditory-visual stimulus onset asynchrony as well as a matched control condition. The PET experiment was designed for both paired image subtraction and correlational analysis methods. Brain regions specifically involved in temporal synchrony-asynchrony detection process were postulated where regional cerebral blood flow (rCBF) responses during task performance are significantly higher than during the control condition and appropriately modulated as a function of increasing task demand. MATERIALS AND METHODSSubjects. T welve right-handed healthy volunteers (seven men, five women, ages 27-56 years) participated in behavioral and PET experiments after giving written informed consent.Behavioral task s. Subjects' ability to detect intermodal temporal mismatch between simple stationary auditory and visual st...

Neural correlates of audiovisual speech processing in a second language

“…In some previous studies, as in the case presented herein, the bilateral pSTS particularly failed to respond to congruency manipulation (Bushara et al, 2001;Miller and D'Esposito, 2005;Ojanen et al, 2005). In addition, other studies found a higher BOLD response to the incongruent condition vs. the congruent condition (Benoit et al,2010 ;Pekkola et al, 2006), or vice versa (Calvert et al, 2000;van Atteveldt et al, 2004;van Atteveldt et al, 2009; note that Calvert et al found only left pSTS activation).…”

Cross-modal integration in a dart-poison frog

“…3). This system affords the possibility of potential future studies of cross-modal integration both behaviorally and neurophysiologically, topics that have received increased attention of late (5,(23)(24)(25)(26)(27)(28).…”

Audiovisual Temporal Correspondence Modulates Human Multisensory Superior Temporal Sulcus Plus Primary Sensory Cortices

“…Atteveldt et al (2006) varied temporal alignment plus semantic congruency between visual letter symbols and auditory phonemes, reporting effects in anterior STS, but their paradigm did not assess crossmodal relationships in rapid temporal patterning (i.e., their letters did not correspond in temporal structure to their speech sounds). Several previous imaging studies (Bushara et al, 2001;Bischoff et al, 2007;Dhamala et al, 2007) used tasks explicitly requiring subjects to judge temporal audiovisual (a)synchrony versus some other task but may thereby have activated task-related networks rather than stimulus-driven modulation as here. Although our results converge with a wide literature in implicating human STS in audiovisual interactions, they go beyond this in showing specifically contralateral activations, determined solely by audiovisual temporal correspondence for nonsemantic stimuli, while identifying interregional functional coupling.…”