Discrete neural substrates underlie complementary audiovisual speech integration processes

Abstract

The ability to combine information from multiple sensory modalities into a single, unified perceptis a key element in an organism’s ability to interact with the external world. This process of perceptual fusion, the binding of multiple sensory inputs into a perceptual gestalt, is highly dependent on the temporal synchrony of the sensory inputs. Using fMRI, we identified two anatomically distinct brain regions in the superior temporal cortex, one involved with processing temporal-synchrony, and one with processing perceptual fusion of audiovisual speech. This dissociation suggests that the superior temporal cortex should be considered a “neuronal hub” comprised of multiple discrete subregions that underlie an array of complementary low-and high -level multisensory integration processes. In this role, abnormalities in the structure and function of superior temporal cortex provide a possible common etiology for temporal-processing and perceptual -fusion deficits seen in a number of clinical populations, including individuals with autism spectrum disorder, dyslexia, and schizophrenia.

Using Neuroeducation as a Model to Evaluate the Effect of Imagery on Chinese Character Writing

“…For the characters memorized in the verbal-contextual method, the participants intended to generate fine-grained images for complete meaning relevant to the character structure. This is consistent with recent neuroimaging studies which found that using verbal-contextual messages may invoke audiovisual integration to form multilayered cross-modal concepts (Campbell, 2008;Stevenson, VanDerKlok, Pisoni, & James, 2011). One important finding was that the verbal-contextual method played an additional function in visualizing the "placement of the scene", which could be detailed image features with spatial connection.…”

Dissociable Effects of Aging and Mild Cognitive Impairment on Bottom-Up Audiovisual Integration

“…A decreased ability of amnestic MCI patients to effectively integrate audiovisual speech information under weak discrepant stimulus conditions can be interpreted within a framework developed through neuroimaging studies investigating the role of the STS in audiovisual integration [33,34,39]. Successful integration of audiovisual stimuli is associated with dynamic changes in functional connectivity between STS and the primary sensory areas that reflect the specific processing demands of different audiovisual stimuli [40,42].…”

“…First, the superior temporal sulcus (STS) has been identified as a critical heteromodal hub for integrating auditory and visual information into a coherent percept [33][34][35][36][37][38][39], with successful integration associated with dynamic changes in the functional connectivity between STS and those visual and auditory cortical sensory areas involved in the initial processing of the stimulus [40][41][42]. Second, baseline volumetric measures of the STS have been found to identify individuals with mild memory impairments who will convert to AD within three years [43], and patients with MCI have shown accelerated widening of the STS over a two-year period compared to stable elderly controls [44].…”

Neural correlates of audiovisual speech processing in a second language

“…The pSTS did not respond during trials when AV information was not perceived to be a fused object, but it displayed activity even for asynchronous stimuli, which were nevertheless perceived as fused. Furthermore, disruption with single-pulse transcranial magnetic stimulation in the pSTS affects AV integration (Beauchamp et al, 2010) Multisensory responses in the pSTS have been shown for speech input at the semantic level (Beauchamp, 2005;Calvert et al, 2000;Stevenson et al, 2010), the phonological level (e.g., non -words: Miller and D´Esposito, 2005;letters: van Atteveldt et al, 2010) and timevarying stimulation with non-speech stimuli (e.g., sinusoidal visual motion aligned with sinusoidally modulated sounds; Werner and Noppeney, 2010;Bischoff et al, 2007). These latter studies have demonstrated that the AV integration system in the pSTS is not languagespecific, but is responsive to AV correspondence in some important features in the speech signal.…”

“…These latter studies have demonstrated that the AV integration system in the pSTS is not languagespecific, but is responsive to AV correspondence in some important features in the speech signal. Furthermore, activation in this region is dominantly bilateral (van Attenveld et al, 2010;Werner and Noppeney, 2010;Bischoff et al, 2007;Beauchamp, 2005), although some studies have also reported unilateral left (Miller and D´Esposito;2005;Calvert et al, 2000) or right (Stevenson et al, 2010) pSTS activation. Indeed, the left and right STS might be functionally different; for example, Calvert et al (2000) suggested that the left and right STS might be involved in speech and non-speech stimuli, respectively.…”