Hand Gestures Have Predictive Potential During Conversation: An Investigation of the Timing of Gestures in Relation to Speech

Abstract

During face‐to‐face conversation, transitions between speaker turns are incredibly fast. These fast turn exchanges seem to involve next speakers predicting upcoming semantic information, such that next turn planning can begin before a current turn is complete. Given that face‐to‐face conversation also involves the use of communicative bodily signals, an important question is how bodily signals such as co‐speech hand gestures play into these processes of prediction and fast responding. In this corpus study, we found that hand gestures that depict or refer to semantic information started before the corresponding information in speech, which held both for the onset of the gesture as a whole, as well as the onset of the stroke (the most meaningful part of the gesture). This early timing potentially allows listeners to use the gestural information to predict the corresponding semantic information to be conveyed in speech. Moreover, we provided further evidence that questions with gestures got faster responses than questions without gestures. However, we found no evidence for the idea that how much a gesture precedes its lexical affiliate (i.e., its predictive potential) relates to how fast responses were given. The findings presented here highlight the importance of the temporal relation between speech and gesture and help to illuminate the potential mechanisms underpinning multimodal language processing during face‐to‐face conversation.

Gestures speed up responses to questions

“…In face-to-face conversation, people rapidly take turns and use hand gestures to communicate. This study provides the first experimental evidence that these gestures speed up responses to questions, in line with corpus work showing that questions with gestures get faster responses (Holler et al, 2018 ; ter Bekke et al, 2024 ) and the multimodal facilitation hypothesis stating that communicative bodily signals facilitate fast language processing (Drijvers & Holler, 2022 ; Holler & Levinson, 2019 ). Despite limited cognitive capacities and strong time pressure (in conversation, and likewise in our experiment), seeing iconic hand gestures facilitates fast responding.…”

“…The mechanism underlying this gestural facilitation may be gestures providing semantic information (Dargue et al, 2019 ; Dargue & Sweller, 2018 ; Holler et al, 2018 ) or increasing attention to speech (Dargue et al, 2019 ; Holler et al, 2018 ). In this study, it appears unlikely that gestures sped up responses by facilitating prediction of upcoming words (Holler & Levinson, 2019 ; ter Bekke et al, 2024 ; Zhang et al, 2021 ). Although most gestures started before their lexical affiliates, similar to natural conversation (Donnellan et al, 2022 ; ter Bekke et al, 2020 ; ter Bekke et al, 2024 ), gestures did not speed up responses more when gestures occurred earlier.…”

“…Correlational corpus data favour accounts postulating a facilitative role of visual signals: questions with hand gestures get faster responses than questions without (Holler et al, 2018 ; ter Bekke et al, 2024 ). However, no causal claims can be made based on corpus data, making it unclear if gestures speed up responses, or a confounding variable causes the effect.…”

“…Responding in conversation requires comprehension of the incoming utterance and simultaneous response planning, after which the response is launched when turn-final cues are detected (Bögels et al, 2015 ; Levinson, 2016 ; Levinson & Torreira, 2015 ). Gestures could facilitate comprehension and response planning by, for example, adding semantic information that helps grasp the utterance’s meaning (Dargue et al, 2019 ; Holler et al, 2018 ), increasing attention to speech (Dargue et al, 2019 ; Holler et al, 2018 ), or improving prediction of upcoming words (Holler & Levinson, 2019 ; ter Bekke et al, 2024 ; Zhang et al, 2021 ). Subsequently, gestures could result in responses being launched earlier by gesture retractions signalling upcoming turn ends (Duncan, 1972 ; Holler et al, 2018 ; Levinson & Torreira, 2015 ).…”

“…Furthermore, if gestures speed up responses by improving predictions about upcoming words, then gestures that precede their corresponding word in speech more (i.e. more predictive potential) may speed up responses more (ter Bekke et al, 2024 ).…”