Iconic gesture and speech integration in younger and older adults

Abstract

This is the accepted version of the paper.This version of the publication may differ from the final published version. This study investigated the impact of age on iconic gesture and speech integration. The performance of a group of older adults (60-76 years) and a group of younger adults (22-30 years) were compared on a task which required the comprehension of information presented in 3 different conditions: verbal only, gesture only, and verbal and gesture combined. The older adults in the study did not benefit as much from multi-modal input as the younger adults and were more likely to ignore gesture when decoding the multi-modal information. Permanent repository link:

Efficiency of speech and iconic gesture integration for robotic and human communicators - a direct comparison

“…There was indeed neither a significant main effect for presenter (F (1, 21) = 2.61, M SEf f ect = 0.01; M SError = 0.004, p = 0.12), nor a significant In order to give an estimate of the effect size of the VG condition, and confirm the results of the pairwise comparisons of the integration scores, we estimated the change in probability of integration target choices (ITC) between unimodal and multimodal conditions, termed the multimodal gain (MMG). Following the method described in Cocks et al [7] this is estimated as the difference between the proportion of ITC in the VG conditions (P(Multi)) and an estimate of the proportion of ITC in unimodal communication (P(Uni)), as in (1).…”

“…To avoid confounds such as the ones potentially inherent in the approaches described above, we decided to base our experiments on a seminal study presented by Cocks et al [7]. In their study, participants were presented with a series of actions conveyed either through speech alone, speech accompanied by an iconic (manner) gesture, or the gesture alone, and asked to choose the appropriate action from a set of action images.…”

“…Stimuli for the experiment consisted of a set of 10 verb phrases, depicting common actions (e.g., I paid, I read), chosen from among those used in the Cocks et al study [7]. Each verb phrase was performed twice, each time accompanied by a different iconic (manner) gesture that conveyed how the action was carried out.…”

“…To match conditions as closely as possible, the same actor is used for human stimuli recorded on video, and to produce the robot stimuli. Given the previous findings of high recognition and integration rates (close to ceiling) using video stimuli of human performers for the type of iconic gestures used in our study [7], we reasoned videos of human gestures would be comparable in efficacy to live performances, with the advantage of being identical across participants.…”

“…Also, multimodal communication is more efficient and effective at conveying information between speaker and listener than unimodal communication [5]. However, in order for gestures to add communicative information to the speech they accompany, information contained in both modes of the communication must be integrated by the listener [5] [7]. This integration process is evidenced by an increase in listener understanding of the information conveyed by the speaker, relative to unimodal communication.…”