A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents

Abstract

Embodied Conversational Agents (ECA) take on different forms, including virtual avatars or physical agents, such as a humanoid robot. ECAs are often designed to produce nonverbal behaviour to complement or enhance its verbal communication. One form of nonverbal behaviour is co-speech gesturing, which involves movements that the agent makes with its arms and hands that is paired with verbal communication. Co-speech gestures for ECAs can be created using different generation methods, such as rule-based and data-driven processes. However, reports on gesture generation methods use a variety of evaluation measures, which hinders comparison. To address this, we conducted a systematic review on co-speech gesture generation methods for iconic, metaphoric, deictic or beat gestures, including their evaluation methods. We reviewed 22 studies that had an ECA with a human-like upper body that used co-speech gesturing in a social human-agent interaction, including a user study to evaluate its performance. We found most studies used a within-subject design and relied on a form of subjective evaluation, but lacked a systematic approach. Overall, methodological quality was low-to-moderate and few systematic conclusions could be drawn. We argue that the field requires rigorous and uniform tools for the evaluation of co-speech gesture systems. We have proposed recommendations for future empirical evaluation, including standardised phrases and test scenarios to test generative models. We have proposed a research checklist that can be used to report relevant information for the evaluation of generative models as well as to evaluate co-speech gesture use.

A Large, Crowdsourced Evaluation of Gesture Generation Systems on Common Data: The GENEA Challenge 2020

“…Human gesture perception is highly subjective, and there are currently no widely accepted objective measures of gesture perception, so most publications have conducted human assessments instead. However, previous subjective evaluations, as reviewed in [59], have several drawbacks, with major ones being the coverage of systems being compared and the scale of the studies. Like in [2,30,31,45], proposed models are at most compared to one or two prior approaches (often a highly similar baseline) or possibly only to ablated versions of the same model.…”

HEMVIP: Human Evaluation of Multiple Videos in Parallel

“…[6,12,21]) and pairwise preference tests (cf. [2,13]) are commonplace; see [20] for a comprehensive review. In MOS tests, participants rate individual stimuli (in our case, videos) on a discrete scale, e.g., 1 through 5.…”

Towards Culture-Aware Co-Speech Gestures for Social Robots

“…After that, the evaluation of the proposed work has been performed relying on subjective evaluation metrics: in particular, a questionnaire based on [45] and [46] has been developed to assess how the cultural background of the user may impact on the way in which the robot's gestures are perceived (Sect. 4.2).…”

“…It must be here pointed out that one of the main problems emerged in similar research works is the difficulty of finding evaluation metrics, i.e. quantitative measurements to assess the naturalness or dynamism of the generated gestures [45]. However, the average jerk of keypoints is usually considered in Literature as a reliable indicator for evaluating autonomous generated gestures [16,45,47].…”