Intelligibility of audiovisual sentences drives multivoxel response patterns in human superior temporal cortex

Abstract

Regions of the human posterior superior temporal gyrus and sulcus (pSTG/S) respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech, and neural responses in pSTG/S may underlie the perceptual benefit of visual speech for the comprehension of noisy auditory speech. We examined this possibility through the lens of multivoxel pattern responses in pSTG/S. BOLD fMRI data was collected from 22 participants presented with speech consisting of English sentences presented in five different formats: visual-only; auditory with and without added auditory noise; and audiovisual with and without auditory noise. Participants reported the intelligibility of each sentence with a button press and trials were sorted post-hoc into those that were more or less intelligible. Response patterns were measured in regions of the pSTG/S identified with an independent localizer. Noisy audiovisual sentences with very similar physical properties evoked very different response patterns depending on their intelligibility. When a noisy audiovisual sentence was reported as intelligible, the pattern was nearly identical to that elicited by clear audiovisual sentences. In contrast, an unintelligible noisy audiovisual sentence evoked a pattern like that of visual-only sentences. This effect was less pronounced for noisy auditory-only sentences, which evoked similar response patterns regardless of intelligibility. The successful integration of visual and auditory speech produces a characteristic neural signature in pSTG/S, highlighting the importance of this region in generating the perceptual benefit of visual speech.

Multivariate fMRI responses in superior temporal cortex predict visual contributions to, and individual differences in, the intelligibility of noisy speech

“…incorrectly encoded items (that were otherwise very similar) were post hoc sorted into separate bins. This experimental design has been successfully applied in studies of noisy speech perception ( Bishop and Miller, 2009 ; Rennig and Beauchamp, 2022 ) and was used in the present study. Noisy speech stimuli were presented and participants reported their perception.…”

“…Using univariate analyses in which one brain location was analyzed at a time, previous neuroimaging studies of individual differences in noisy speech perception found conflicting results ( Du et al, 2016 ; Kim et al, 2021 ; McGettigan et al, 2012b ). One reason may be that univariate measures have limited power; in a direct comparison within individuals, Rennig and Beauchamp (2022) found that neural differences between intelligible and unintelligible speech were much larger for multivariate (χ 2 = 102) than univariate (χ 2 = 11) analyses.…”

“…The fMRI dataset for the word study has not been previously published. The fMRI dataset for the sentence study is a re-analysis of published data ( Rennig and Beauchamp, 2022 ).…”

“…For multivariate fMRI analysis, the mean percentage signal change across conditions in each surface node was calculated and subtracted from the response to each individual condition. This increases the dynamic range of the fMRI pattern correlation ( Haxby et al, 2001 ) and is especially important in superior temporal cortex where many nodes show a positive response to all speech stimuli ( Rennig and Beauchamp, 2022 ). To compute the fMRI pattern similarity between each pair of conditions, the mean-centered percentage signal change across the ROI for the first condition was correlated with the mean-centered percentage signal change in the second condition, resulting in a single correlation value for each pair of conditions for each hemisphere.…”

Age-Related Changes to Multisensory Integration and Audiovisual Speech Perception

“…From a neurobiological perspective, the STS is a clear target for research into audiovsiual integration due to its location at the junction between occipital and temporal cortex [ 32 ]. Neurons within the STS display strong activation during the processing of meaningful everyday visual and auditory stimuli, such as moving people or objects, or speech and language comprehension [ 78 , 128 , 129 , 130 , 131 , 132 ]. Activity in the STS also reflects the “super-additivity” theory in multisensory integration, in that bimodal stimuli elicit a greater neural response in this brain area than when unisensory stimuli are presented [ 128 ] (however, see [ 133 ]).…”