A Comprehensive Review of Data-Driven Co-Speech Gesture Generation

Abstract

Figure 1: Co-speech gesture generation approaches can be divided into rule-based and data-driven. Rule-based systems use carefully designed heuristics to associate speech with gesture (Section 4). Data-driven approaches associate speech and gesture through statistical modeling (Section 5.2), or by learning multimodal representations using deep generative models (Section 5.3). The main input modalities are speech audio in an intermediate representation; text transcript of speech; humanoid pose in joint position or angle form; and control parameters for motion design intent. Virtual agents and social robotics are the main research applications, although also compatible with games and film VFX.

UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons

“…We found the hand/finger quality of the existing mocap datasets [56] is not good enough, especially when retargeted to an avatar. Datasets claimed with high-quality hand motion capture were still reported to have poor hand motion [56], e.g., ZEGGS Dataset in [5] and Talking With Hands in [90]. So we ignore hand/finger motion currently, and leave it to future work.…”

“…We perform the training and evaluation on the Trinity [16] and ZEGGS [19] datasets. Even based on motion capture, the hand quality is still low [5,56,90], so we ignore hand motion currently. Then the number of joints for the two datasets is 𝐽 𝐴 = 26 and 𝐽 𝐵 = 27, respectively.…”

“…Nonverbal behaviors, including gestures, play key roles in conveying messages in human communication [37]. The automatic co-speech gesture generation is considered an enabling technology to create realistic 3D avatars in films, games, virtual social spaces, and for interaction with social robots [56]. In the era of deep learning, existing data-driven gesture generation methods usually rely on a large dataset.…”

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

“…To alleviate the manual effort of designing rules in rule-based methods, data-driven approaches have gradually become predominant in this field. Nyatsanga et al [2023] offer a thorough survey of these methods. Early data-driven approaches aim to directly learn mapping rules from data through statistical models [Levine et al 2010[Levine et al , 2009Neff et al 2008] and combine them with predefined gesture units for gesture generation.…”

GesGPT: Speech Gesture Synthesis With Text Parsing from GPT

“…As shown in the analysis of GENEA Challenge 2022 [15], the generated gestures have exhibited superior human-like similarity compared to motion capture data; nevertheless, enhancing the semantic expressiveness of these gestures remains an area requiring further exploration. Secondly, the deep learning-based methods for gesture synthesis tend to produce average results, often failing to generate nuanced and intricate hand movements [16]. Lastly, they may not adequately address the inherent intention in gesture expressions, which can lead to the generation of less plausible or contextually inappropriate gestures.…”