Speech comprehension aided by multiple modalities: Behavioural and neural interactions

Abstract

Speech comprehension is a complex human skill, the performance of which requires the perceiver to combine information from several sources - e.g. voice, face, gesture, linguistic context - to achieve an intelligible and interpretable percept. We describe a functional imaging investigation of how auditory, visual and linguistic information interact to facilitate comprehension. Our specific aims were to investigate the neural responses to these different information sources, alone and in interaction, and further to use behavioural speech comprehension scores to address sites of intelligibility-related activation in multifactorial speech comprehension. In fMRI, participants passively watched videos of spoken sentences, in which we varied Auditory Clarity (with noise-vocoding), Visual Clarity (with Gaussian blurring) and Linguistic Predictability. Main effects of enhanced signal with increased auditory and visual clarity were observed in overlapping regions of posterior STS. Two-way interactions of the factors (auditory × visual, auditory × predictability) in the neural data were observed outside temporal cortex, where positive signal change in response to clearer facial information and greater semantic predictability was greatest at intermediate levels of auditory clarity. Overall changes in stimulus intelligibility by condition (as determined using an independent behavioural experiment) were reflected in the neural data by increased activation predominantly in bilateral dorsolateral temporal cortex, as well as inferior frontal cortex and left fusiform gyrus. Specific investigation of intelligibility changes at intermediate auditory clarity revealed a set of regions, including posterior STS and fusiform gyrus, showing enhanced responses to both visual and linguistic information. Finally, an individual differences analysis showed that greater comprehension performance in the scanning participants (measured in a post-scan behavioural test) were associated with increased activation in left inferior frontal gyrus and left posterior STS. The current multimodal speech comprehension paradigm demonstrates recruitment of a wide comprehension network in the brain, in which posterior STS and fusiform gyrus form sites for convergence of auditory, visual and linguistic information, while left-dominant sites in temporal and frontal cortex support successful comprehension.

Prediction and constraint in audiovisual speech perception

“…However, it may also be that posterior STS is active in determining relative weighting or attention directed at complementary modalities. Consistent with this view, activity in posterior STS also appears to distinguish between variations in visual clarity to a greater degree when speech is less intelligible (McGettigan et al, 2012). 3 …”

Narrowed Expectancies under Degraded Speech: Revisiting the N400

“…Although a whole tradition of behavioral studies have laid the ground work for understanding cognitive processes in adverse listening conditions (Mattys et al, 2009;Davis & Johnsrude, 2003;Pichora-Fuller, 2003;Stickney & Assmann, 2001;Kalikow et al, 1977;Miller et al, 1951), only a few neuroimaging studies (e.g., McGettigan et al, 2012;Davis, Ford, Kherif, & Johnsrude, 2011;Obleser & Kotz, 2010;Obleser, Wise, Alex Dresner, & Scott, 2007) and EEG studies (e.g., Boulenger, Hoen, Jacquier, & Meunier, 2011;Obleser & Kotz, 2011;Romei, Wambacq, Besing, Koehnke, & Jerger, 2011;Aydelott, Dick, & Mills, 2006;Connolly, Phillips, Stewart, & Brake, 1992) have taken on the issue of semantic or expectancy benefits in degraded speech.…”

Audiovisual Integration in Children Listening to Spectrally Degraded Speech

“…For example, when the auditory and visual information do not match, auditory stimuli can be misperceived, as in the well-documented McGurk effect (McGurk & MacDonald, 1976): An auditory signal /ba / simultaneously presented with a visual /ga / often results in an illusionary percept /da /. Furthermore, a large body of empirical work has shown that viewing a speaker's mouth movements provides additional information that can improve auditory speech perception, particularly when the auditory signal is masked by background sounds (e.g., Bishop & Miller, 2009;McGettigan et al, 2012;Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007;Sánchez-García, Alsius, Enns, & Soto-Faraco, 2011;Summerfield, MacLeod, McGrath, & Brooke, 1989).…”