The Effects of Iconic Gestures and Babble Language on Word Intelligibility in Sentence Context

Abstract

Purpose: This study investigated to what extent iconic co-speech gestures help word intelligibility in sentence context in two different linguistic maskers (native vs. foreign). It was hypothesized that sentence recognition improves with the presence of iconic co-speech gestures and with foreign compared to native babble. Method: Thirty-two native Dutch participants performed a Dutch word recognition task in context in which they were presented with videos in which an actress uttered short Dutch sentences (e.g., Ze begint te openen, “She starts to open”). Participants were presented with a total of six audiovisual conditions: no background noise (i.e., clear condition) without gesture, no background noise with gesture, French babble without gesture, French babble with gesture, Dutch babble without gesture, and Dutch babble with gesture; and they were asked to type down what was said by the Dutch actress. The accurate identification of the action verbs at the end of the target sentences was measured. Results: The results demonstrated that performance on the task was better in the gesture compared to the nongesture conditions (i.e., gesture enhancement effect). In addition, performance was better in French babble than in Dutch babble. Conclusions: Listeners benefit from iconic co-speech gestures during communication and from foreign background speech compared to native. These insights into multimodal communication may be valuable to everyone who engages in multimodal communication and especially to a public who often works in public places where competing speech is present in the background.

Children benefit from gestures to understand degraded speech but to a lesser extent than adults

“…Secondly, this study revealed that not only adults but also children can benefit from iconic gestures to disambiguate degraded speech. Likewise, this finding is consistent with previous studies with adults revealing that iconic gestures help adult listeners to comprehend information conveyed by speech in noise (Drijvers and Özyürek, 2017;Schubotz et al, 2020;Wilms et al, 2022). This finding is also consistent with previous studies which showed that children can integrate gestures with clear speech (e.g., Kelly and Church, 1998;Church et al, 2000;Kelly, 2001;Broaders and Goldin-Meadow, 2010;Sekine et al, 2015Sekine et al, , 2020.…”

“…Previous studies have shown that gestures, defined as meaningful hand movements that accompany speech, are part of an integrated system of language ( McNeill, 1992 ; Kita and Özyürek, 2003 ; Özyürek, 2014 ), both in production and comprehension ( Kelly et al, 2010 ), and both in typical environments and in noisy situations. In line with this integrated view of speech and gesture, it has been also shown that adult listeners are flexible and use gestures to disambiguate comprehending speech with externally- or internally-induced noise, as observed in degraded speech (e.g., Drijvers and Özyürek, 2017 , 2020 ; Schubotz et al, 2020 ; Wilms et al, 2022 ), in noisy environments ( Kendon, 2004 ), or in instances involving hearing difficulties ( Obermeier et al, 2012 ). It is not clear, however, whether this flexible integrated system is in place in children as in adults, especially in noisy situations which are more taxing than in clear speech situations.…”

“…For example, it has been observed that people effectively use and modulate their gestures in adverse listening conditions (Kendon, 2004;Trujillo et al, 2021). Numerous studies have empirically shown that the use of iconic gestures is beneficial for adult listeners when speech is degraded as well (e.g., Rogers, 1978;Holle et al, 2010;Drijvers and Özyürek, 2017;Wilms et al, 2022). For example, Drijvers and Özyürek (2017) examined the enhancement effect of both iconic gestures and lip movements (visible speech) on the comprehension of degraded speech by comparing comprehension among three different noise level conditions: 2-band noise-vocoding (severe noise), 6-band noise-vocoding (moderate noise) and clear speech.…”

Environmental noise affects audiovisual gain during speech comprehension in adverse listening conditions

“…As a first step, we established whether we could replicate the gestural enhancement effect for degraded speech comprehension (as observed in e.g., Drijvers & Ozyurek, 2017;Wilms et al, 2022). For this purpose, we fitted a linear mixed effects model using lme4 (version; Bates et al, 2015) in R (4.2.3, R Core Team, 2023), and obtained P-values with the lmerTest (3.1.1, Kuznetsova et al, 2017) package.…”

Children Benefit from Gestures to Understand Degraded Speech but to a Lesser Extent than Adults

“…Numerous studies have empirically shown that the use of iconic gestures is bene cial for adult listeners when speech is degraded as well (e.g. Drijvers & Özyürek, 2017;Holle, Obleser, Rueschemeyer, & Gunter, 2010;Roger, 1978;Wilms et al, 2022). For example, Drijvers and Özyürek (2017) examined the enhancement effect of both iconic gestures and lip movements (visible speech) on the comprehension of degraded speech by comparing comprehension among three different noise level conditions: 2-band noise-vocoding (severe noise), 6-band noisevocoding (moderate noise) and clear speech.…”