When Language Meets Action: The Neural Integration of Gesture and Speech

Abstract

Although generally studied in isolation, language and action often co-occur in everyday life. Here we investigated one particular form of simultaneous language and action, namely speech and gestures that speakers use in everyday communication. In a functional magnetic resonance imaging study, we identified the neural networks involved in the integration of semantic information from speech and gestures. Verbal and/or gestural content could be integrated easily or less easily with the content of the preceding part of speech. Premotor areas involved in action observation (Brodmann area [BA] 6) were found to be specifically modulated by action information "mismatching" to a language context. Importantly, an increase in integration load of both verbal and gestural information into prior speech context activated Broca's area and adjacent cortex (BA 45/47). A classical language area, Broca's area, is not only recruited for language-internal processing but also when action observation is integrated with speech. These findings provide direct evidence that action and language processing share a high-level neural integration system.

Event-related brain potentials during the monitoring of speech errors

“…pictures, line drawings, or gestures) into a context have reported a more frontal scalp distribution for the N400 effects (e.g. Federmeier, and Kutas, 2001;Ganis, Kutas, and Sereno, 1996;Holle, and Gunter, 2007;Willems et al, 2007). Here, it is suggested that in Experiment 2 the additional attempt to predict upcoming words involves (parts of) the language production system (Pickering, and Garrod, 2007).…”

Neural evidence for the interplay between language, gesture, and action: A review

“…For instance, in the case of language comprehension, it has become increasingly clear that the brain uses several types of information in a qualitatively similar way to arrive at a full understanding of a message. This includes information from world-knowledge, co-speech gestures, pictures, speaker's identity derived from voice characteristics and information from a preceding discourse (Federmeier & Kutas, 2001;Hagoort, 2005;Hagoort, Hald, Bastiaansen, & Petersson, 2004;Hagoort & van Berkum, in press;Nieuwland & van Berkum, 2006;Ö zyü rek et al, in press;van Berkum, Hagoort, & Brown, 1999;van Berkum, Zwitserlood, Hagoort, & Brown, 2003;Willems et al, 2006). Importantly, these examples serve to demonstrate that the brain not only is capable of taking several streams of information into account, but actually does so in a qualitatively similar way.…”

“…However, a remaining question is how comparable the semantic processing evoked by hand gestures is to that of linguistic items such as words. This was investigated in a pair of studies measuring the neural time course (using ERPs) as well as the neural locus (using fMRI) of the sentence level integration of co-speech gestures and spoken words (Ö zyü rek, Willems, Kita, & Hagoort, in press;Willems, Ö zyü rek, & Hagoort, 2006). Subjects heard sentences in which a critical word was accompanied by a gesture.…”

“…In other words, these gestures partly drive the neural language system, but also lead to specific effects not observed for spoken language. An example of this is the modulation of premotor areas when gestures are not in accordance with a previous sentence context (Willems et al, 2006).…”

Does comprehension of symbolic gestures and corresponding-in-meaning words make use of motor simulation?

“…Villarreal and colleagues [5] assessed cortical activity during recognition of communicative gestures containing symbolic connotations (e.g., The embodied theory of language assumes that language comprehension makes use of the neural system ordinarily recruited for action control [6]. Focusing on spoken language material related to concrete actions, recent neurophysiological studies have shown that premotor regions are involved in language processing [7]. Also, in keeping with the involvement of the motor system in processing action-related material, the results reported by Buccino et al [8] in a single pulse TMS study, have shown that motor evoked potentials (MEPs) recorded from hand muscles are modulated during listening to hand-related action sentences.…”