Filling in the gaps: observing gestures conveying additional information can compensate for missing verbal content

Abstract

While observing gesture has been shown to benefit narrative recall and learning, research has yet to show whether gestures that provide information that is missing from speech benefit narrative recall. This study explored whether observing gestures that relay the same information as speech and gestures that provide information missing from speech differentially affect narrative recall in university students. Participants were presented with a videotaped narrative told in one of four conditions: with gestures and no missing verbal information, with gestures and missing verbal information, with no gestures and no missing verbal information, or with no gestures and missing verbal information. Results showed that observing gestures that provided additional information to speech (i.e., when the speech was missing vital information) enhanced narrative recall compared to observing no gestures, while observing gestures that did not provide additional information to speech were no more beneficial than observing no gestures at all. Findings from the current study provide valuable insight into the beneficial effect of iconic gesture on narrative recall, with important implications for education and learning.

Speech-In-Noise Comprehension is Improved When Viewing a Deep-Neural-Network-Generated Talking Face

“…Future work should analyze how well this model works with other languages. Second, in addition to the moving lips and faces that provide the visual speech cues, non-verbal gestures and other spontaneous movements (blinking, for example) also have an impact on normal communication ( Cassell et al, 1999 ; Dargue et al, 2021 ; Goldin-Meadow, 1999 ). A model that takes the head movement into account has been developed ( Chen et al, 2020 ) that allows a potential future investigation on what impact of such model can bring in speech comprehension.…”