Co‐speech gestures influence neural activity in brain regions associated with processing semantic information

Abstract

Everyday communication is accompanied by visual information from several sources, including cospeech gestures, which provide semantic information listeners use to help disambiguate the speaker's message. Using fMRI, we examined how gestures influence neural activity in brain regions associated with processing semantic information. The BOLD response was recorded while participants listened to stories under three audiovisual conditions and one auditory-only (speech alone) condition. In the first audiovisual condition, the storyteller produced gestures that naturally accompany speech. In the second, she made semantically unrelated hand movements. In the third, she kept her hands still. In addition to inferior parietal and posterior superior and middle temporal regions, bilateral posterior superior temporal sulcus and left anterior inferior frontal gyrus responded more strongly to speech when it was further accompanied by gesture, regardless of the semantic relation to speech. However, the right inferior frontal gyrus was sensitive to the semantic import of the hand movements, demonstrating more activity when hand movements were semantically unrelated to the accompanying speech. These findings show that perceiving hand movements during speech modulates the distributed pattern of neural activation involved in both biological motion perception and discourse comprehension, suggesting listeners attempt to find meaning, not only in the words speakers produce, but also in the hand movements that accompany speech.Keywords discourse comprehension; fMRI; gestures; semantic processing; inferior frontal gyrus Face-to-face communication is based on more than speech alone. Audible speech is only one component of a communication system that also includes co-speech gestures-hand and arm movements that accompany spoken language (Kendon, 1994;McNeill, 1992; McNeill, 2005). Such co-speech gestures serve an important role in face-to-face communication for both speaker and listener. Listeners not only process the words that speakers produce, but also continuously integrate gestures with speech and with other visual information (e.g., the speaker's lips, mouth, and eyes) to arrive at the speaker's meaning (Goldin-Meadow, 2006;Kendon, 1994; McNeill, 2005). Despite the importance of co-speech gesture to communicative

A common functional neural network for overt production of speech and gesture

“…It seems reasonable that cospeech gesture production engages the gesture network because it combines conceptual, as well as skill-related aspects of actions that are essential for the meaningful hand movements of co-speech gesturing. Importantly, the second activity pattern includes fronto-parietal areas that are also frequently found in neuroimaging studies of co-speech gesture perception (Dick et al, 2009;Holle et al, 2008;Kircher et al, 2009;Straube et al, 2011;Willems et al, 2007Willems et al, , 2009). …”

“…Their coordination with speech is both temporal-kinetic and semantic (Kita & Özyürek, 2003;Loehr, 2007) and they have been shown to affect learning and memory in the listener as well as the speaker (Goldin-Meadow, 2003;Hostetter, 2011;Marstaller & Burianová, 2013). Previous neuroimaging studies have found that the observation of co-speech gestures engages superior and middle temporal gyrus, intraparietal sulcus, and inferior frontal gyrus (Dick et al, 2009;Holle et al, 2008Holle et al, , 2010Hubbard et al, 2009;Kircher et al, 2009;Skipper et al, 2007Skipper et al, , 2009Straube et al, 2011;Willems et al, 2007Willems et al, , 2009). The findings from these studies strongly suggest that during the observation of co-speech gestures, frontal and temporal regions are engaged in semantic processing, whereas frontal and parietal areas are activated for action understanding (Marstaller & Burianová, 2014).…”

Intention Processing in Communication: A Common Brain Network for Language and Gestures

“…Generally, the left ventral premotor BA 6/44 complex is also involved in these task, but the lack of a formal comparison with corresponding linguistic conditions, as we have performed here, does not allow the question of hemispheric dominance for the two modalities to be tackled. Interestingly, recent studies (Dick, Goldin-Meadow, Hasson, Skipper, & Small, 2009;Green et al, 2009;Straube, Green, Weis, Chatterjee, & Kircher, 2009) found that the right inferior frontal gyrus displayed more activity when hand movements were semantically unrelated than when they were related to the accompanying speech. Altogether, these findings may be compatible with the results of our study, in which the participants were required to choose a compatible outcome for the intention revealed by the extralinguistic communicative gesture, and discard the noncompatible outcome.…”

Linking language to the visual world: Neural correlates of comprehending verbal reference to objects through pointing and visual cues

“…The fact that we did not find pSTS activation in the conjunction analysis comparing a bimodal (i.e., speech + pointing) condition to the sum of the unimodal conditions may be due to the absence of motion in our visual stimuli (cf. Dick et al, 2009). The current study may serve as a baseline for future studies investigating the processing of pointing gestures and speech in more dynamic and interactive situations (cf.…”