Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension

Abstract

In face-to-face discourse, listeners exploit cues in the input to generate predictions about upcoming words. Moreover, in addition to speech, speakers produce a multitude of visual signals, such as iconic gestures, which listeners readily integrate with incoming words. Previous studies have shown that processing of target words is facilitated when these are embedded in predictable compared to non- predictable discourses and when accompanied by iconic compared to meaningless gestures. In the present study, we investigated the interaction of both factors. We recorded electroencephalogram from 60 Dutch adults while they were watching videos of an actress producing short discourses. The stimuli consisted of an introductory and a target sentence; the latter contained a target noun. Depending on the preceding discourse, the target noun was either predictable or not. Each target noun was paired with an iconic gesture and a gesture that did not convey meaning. In both conditions, gesture presentation in the video was timed such that the gesture stroke slightly preceded the onset of the spoken target by 130 ms. Our ERP analyses revealed independent facilitatory effects for predictable discourses and iconic gestures. However, the interactive effect of both factors demonstrated that target processing (i.e., gesture-speech integration) was facilitated most when targets were part of predictable discourses and accompanied by an iconic gesture. Our results thus suggest a strong intertwinement of linguistic predictability and non-verbal gesture processing where listeners exploit predictive discourse cues to pre-activate verbal and non-verbal representations of upcoming target words.

The role of co-speech gestures in retrieval and prediction during naturalistic multimodal narrative processing

“…Here, we found that both iconic and beat gestures consistently modulate the frequency-dependent N400; but for surprisal, although the mTRF model fit generally improved when both iconic and beat gesture coding were additionally included as regressors, the pairwise comparison between gesture present and gesture absent words only showed a significant modulation effect of beat gesture on the surprisal-dependent N400. Other recent literature has provided evidence of dissociable effects of gesture types on electrophysiological responses using both naturalistic and factorial approaches, with iconic (i.e., meaningful) gestures being associated to a reduction in the amplitude of the N400 and beat gestures being associated to increased N400 effects as derived from semantic violation or surprisal (Hintz et al, 2022;. On the other hand, there is another line of literature showing that the N400 amplitude of single words in a sentence may still be modulated by the presence of beat gestures, suggesting a potentially facilitative effect of beat gestures (Morett et al, 2020;Wang & Chu, 2013).…”

“…Notably, despite prior research showing mixed effect of beat gestures on semantic processing and potentially differential effect of iconic gestures and other gestures (Hintz et al, 2022;Morett et al, 2020;Wang & Chu, 2013;, for the purpose of testing the facilitative role of gestures in general on semantic processing, we collapsed across all gesture types for all analyses for the main analysis (see Cuevas et al, 2019 for an fMRI study using the same stimuli). We nevertheless reported specific effects of both iconic and beat gestures in the supplement.…”