Audiovisual Temporal Correspondence Modulates Human Multisensory Superior Temporal Sulcus Plus Primary Sensory Cortices

Abstract

The brain should integrate related but not unrelated information from different senses. Temporal patterning of inputs to different modalities may provide critical information about whether those inputs are related or not. We studied effects of temporal correspondence between auditory and visual streams on human brain activity with functional magnetic resonance imaging (fMRI). Streams of visual flashes with irregularly jittered, arrhythmic timing could appear on right or left, with or without a stream of auditory tones that coincided perfectly when present (highly unlikely by chance), were noncoincident with vision (different erratic, arrhythmic pattern with same temporal statistics), or an auditory stream appeared alone. fMRI revealed blood oxygenation level-dependent (BOLD) increases in multisensory superior temporal sulcus (mSTS), contralateral to a visual stream when coincident with an auditory stream, and BOLD decreases for noncoincidence relative to unisensory baselines. Contralateral primary visual cortex and auditory cortex were also affected by audiovisual temporal correspondence or noncorrespondence, as confirmed in individuals. Connectivity analyses indicated enhanced influence from mSTS on primary sensory areas, rather than vice versa, during audiovisual correspondence. Temporal correspondence between auditory and visual streams affects a network of both multisensory (mSTS) and sensory-specific areas in humans, including even primary visual and auditory cortex, with stronger responses for corresponding and thus related audiovisual inputs.

Touch, sound and vision in human superior temporal sulcus

“…Previous studies have shown that posterior STS responds more to multisensory auditory-visual stimuli than to unisensory auditory or visual stimuli (Beauchamp et al, 2004b;Calvert, 2001;Hein et al, 2007;Noesselt et al, 2007;Raij et al, 2000;Van Atteveldt et al, 2004). Consistent with these results, we observed a larger response for multisensory auditory-tactile stimuli than unisensory auditory or tactile stimulation.…”

“…In the banks of human posterior STS, neuroimaging studies have reported multisensory responses to auditory and visual stimulation (Beauchamp et al, 2004b;Calvert, 2001;Noesselt et al, 2007;Van Atteveldt et al, 2004;Wright et al, 2003). This region has been termed STSms, the STS multisensory region (Beauchamp et al, 2004a).…”

Long-term music training tunes how the brain temporally binds signals from multiple senses

“…At the neural level, asynchronous relative to synchronous conditions increased activation in a distributed neural system encompassing bilateral STS/MTG, occipital and fusiform gyri, and premotor and cerebellar cortices (6,7). This asynchrony system was largely shared by speech and music, with no asynchrony effects that were specific to speech or music.…”

“…Although speech elicited comparable asynchrony effects in musicians and nonmusicians in posterior STS/MTG bilaterally and left cerebellum, music evoked increased asynchrony effects for musicians relative to nonmusicians in left premotor cortex, left cerebellum, and right posterior STS/MTG. Hence, audiovisual asynchrony is detected automatically not only along the sensory processing hierarchies and classical audiovisual integration areas such as STS (7,20,21) but also in a premotor-cerebellar circuitry. Importantly, the asynchrony effects within the premotor-cerebellar circuitry depended on the availability of prior sensory-motor experience.…”

“…Given the variability in physical and neural transmission times, sensory signals do not have to be precisely synchronous but must co-occur within a temporal window that flexibly adapts to the temporal statistics of the sensory inputs as a consequence of music (3) or audiovisual training (4). At the neural level, audiovisual (a)synchrony processing relies on a widespread neural system encompassing subcortical, primary sensory, higher-order association, cerebellar, and premotor areas (5)(6)(7)(8).…”