Cortical integration of audio–visual speech and non-speech stimuli

Abstract

We investigated the neural basis of audio-visual processing in speech and non-speech stimuli. Physically identical auditory stimuli (speech and sinusoidal tones) and visual stimuli (animated circles and ellipses) were used in this fMRI experiment. Relative to unimodal stimuli, each of the multimodal conjunctions showed increased activation in largely non-overlapping areas. The conjunction of Ellipse and Speech, which most resembles naturalistic audiovisual speech, showed higher activation in the right inferior frontal gyrus, fusiform gyri, left posterior superior temporal sulcus, and lateral occipital cortex. The conjunction of Circle and Tone, an arbitrary audio-visual pairing with no speech association, activated middle temporal gyri and lateral occipital cortex. The conjunction of Circle and Speech showed activation in lateral occipital cortex, and the conjunction of Ellipse and Tone did not show increased activation relative to unimodal stimuli. Further analysis revealed that middle temporal regions, although identified as multimodal only in the Circle-Tone condition, were more strongly active to Ellipse-Speech or Circle-Speech, but regions that were identified as multimodal for Ellipse-Speech were always strongest for Ellipse-Speech. Our results suggest that combinations of auditory and visual stimuli may together be processed by different cortical networks, depending on the extent to which speech or non-speech percepts are evoked.

Adult dyslexic readers benefit less from visual input during audiovisual speech processing: fMRI evidence

“…Typical and dyslexic readers differed in how much they benefited from visual information during audiovisual processing, with dyslexic readers showing reduced activity in the left fusiform gyrus. The fusiform gyrus is involved in audiovisual speech perception (Stevenson et al, 2010;Wyk et al, 2010) andcomprehension (McGettigan et al, 2012). The difference found between the groups in the fusiform gyrus is in accordance with DTI studies showing white matter differences between typical and dyslexic readers in this region (e.g., Richards et al, 2008).…”

“…1B). For the speech perception conditions, we chose areas well described in the literature as responding to unimodal auditory material: superior temporal gyrus/sulcus and Heschl's gyrus (e.g., Calvert et al, 1999;Callan et al, 2003;Möttönen et al, 2004;Pekkola et al, 2005); to unimodal visual material (more specifically, to movement and faces): the middle temporal gyrus (e.g., Callan et al, 2003;Calvert et al, 1999;Calvert and Campbell, 2003) and fusiform gyrus (e.g., Calvert and Campbell, 2003;Macaluso et al, 2004;Stevenson et al, 2010;Wyk et al, 2010); and to audiovisual material: the superior temporal gyrus/sulcus and supramarginal gyrus (e.g., Beauchamp et al, 2004aBeauchamp et al, , 2010Calvert et al, 2000;Skipper et al, 2005;Stevenson et al, 2010Stevenson et al, , 2011. We additionally included the planum temporale as an area of interest, given previous literature on reduced responsiveness of that area to speech in dyslexia (e.g., Monzalvo et al, 2012).…”

“…In typical readers, a number of brain regions have been repeatedly implicated in audiovisual speech processing. These regions include high-level associative areas such as the superior temporal gyrus (including the planum temporale) (e.g., Beauchamp et al, 2004a;Beauchamp et al, 2010;Calvert et al, 2000;Stevenson et al, 2010;Stevenson et al, 2011) and the supramarginal gyrus (e.g., Skipper et al, 2005), as well as other more primary sensory regions such as Heschl's gyrus (e.g., Calvert et al, 1999;Callan et al, 2003;Möttönen et al, 2004;Pekkola et al, 2005) and the superior temporal gyrus (e.g., Beauchamp et al, 2004aBeauchamp et al, , 2010Calvert et al, 2000) for auditory information, and the middle temporal gyrus (e.g., Callan et al, 2003;Calvert et al, 1999;Calvert and Campbell, 2003) and the fusiform gyrus (e.g., Calvert and Campbell, 2003;Macaluso et al, 2004;Stevenson et al, 2010;Wyk et al, 2010) for visual information processing.…”

“…In auditory speech, differences between typical and dyslexic readers might be found in Heschl's gyrus (e.g., Calvert et al, 1999;Callan et al, 2003;Möttönen et al, 2004;Pekkola et al, 2005) and in the superior temporal gyrus (e.g., Beauchamp et al, 2004aBeauchamp et al, , 2010Calvert et al, 2000). For visual speech, the middle temporal and the fusiform gyri are plausible candidates for group differences (e.g., Callan et al, 2003;Calvert et al, 1999;Calvert and Campbell, 2003;Macaluso et al, 2004;Stevenson et al, 2010;Wyk et al, 2010). Lastly, in audiovisual speech, differences between typical and dyslexic readers could be found in the superior temporal gyrus and in the supramarginal gyrus (Beauchamp et al, 2004a(Beauchamp et al, , 2010Skipper et al, 2005;Stevenson et al, 2010Stevenson et al, , 2011, which are known to integrate audiovisual information, but also in any of the unisensory regions mentioned above.…”

Effects of Auditory Stimuli in the Horizontal Plane on Audiovisual Integration: An Event-Related Potential Study

“…Moreover, Wyk et al (2010) investigated the cortical integration of audiovisual speech and non-speech stimuli. Their findings demonstrated the differential activation of cortical networks involved in the integration of speech and non-speech stimuli [24]. However, whether simple non-speech signals originating from behind participants can enhance the detection of visual stimuli remains unclear.…”