Gesture Facilitates the Syntactic Analysis of Speech

Abstract

Recent research suggests that the brain routinely binds together information from gesture and speech. However, most of this research focused on the integration of representational gestures with the semantic content of speech. Much less is known about how other aspects of gesture, such as emphasis, influence the interpretation of the syntactic relations in a spoken message. Here, we investigated whether beat gestures alter which syntactic structure is assigned to ambiguous spoken German sentences. The P600 component of the Event Related Brain Potential indicated that the more complex syntactic structure is easier to process when the speaker emphasizes the subject of a sentence with a beat. Thus, a simple flick of the hand can change our interpretation of who has been doing what to whom in a spoken sentence. We conclude that gestures and speech are integrated systems. Unlike previous studies, which have shown that the brain effortlessly integrates semantic information from gesture and speech, our study is the first to demonstrate that this integration also occurs for syntactic information. Moreover, the effect appears to be gesture-specific and was not found for other stimuli that draw attention to certain parts of speech, including prosodic emphasis, or a moving visual stimulus with the same trajectory as the gesture. This suggests that only visual emphasis produced with a communicative intention in mind (that is, beat gestures) influences language comprehension, but not a simple visual movement lacking such an intention.

Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

“…associated structural revision. A similar distinction between the P600 and the N400 (reflecting lexical-semantic processes) is also apparent in research on the integration of co-speech gestures (Holle et al, 2012;Kelly, Kravitz, & Hopkins, 2004).…”

“…Another kind of gesture (beat gestures) can affect comprehension processes such as structural disambiguation in the face of temporary linguistic ambiguity (subjectobject, SO compared with object-subject, OS, Holle et al, 2012). A verb following Knoeferle, P. (to appear).…”

Beat gestures modulate auditory integration in speech perception

“…Beats are thought to play the same role as head nods or eyebrows which have been shown to produce improvements in comprehension (Krahmer & Swerts, 2007;Munhall, Jones, Callan, Kuratate & Vatikiotis-Bateson, 2004). Of course, one potential role of these attentional highlighters would be to facilitate the syntactic parsing of the discourse, as it has been found by Holle et al (2012), the only study to date addressing beat gestures combined with ERPs as far as we know. Using sentences with ambiguous low frequency syntactic forms, Holle et al found that beats could modulate the amplitude of the P600 component, a marker of processing cost for syntactic…”

Beat gestures modulate auditory integration in speech perception

“…One recent study has investigated how beat gestures can help the listener disambiguate sentences with a complex syntactic structure using ERPs (Holle et al, 2012). Holle et al showed a modulation of the P600 component (indexing the difficulty of resolving complex syntactic structures) when beats emphasized which noun was the subject of the sentence.…”