Rhythmic Gesticulator

Abstract

Automatic synthesis of realistic co-speech gestures is an increasingly important yet challenging task in artificial embodied agent creation. Previous systems mainly focus on generating gestures in an end-to-end manner, which leads to difficulties in mining the clear rhythm and semantics due to the complex yet subtle harmony between speech and gestures. We present a novel co-speech gesture synthesis method that achieves convincing results both on the rhythm and semantics. For the rhythm, our system contains a robust rhythm-based segmentation pipeline to ensure the temporal coherence between the vocalization and gestures explicitly. For the gesture semantics, we devise a mechanism to effectively disentangle both low- and high-level neural embeddings of speech and motion based on linguistic theory. The high-level embedding corresponds to semantics, while the low-level embedding relates to subtle variations. Lastly, we build correspondence between the hierarchical embeddings of the speech and the motion, resulting in rhythm- and semantics-aware gesture synthesis. Evaluations with existing objective metrics, a newly proposed rhythmic metric, and human feedback show that our method outperforms state-of-the-art systems by a clear margin.

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

“…Due to the inherent many-to-many relationship between speech and gesture, end-to-end models can generate natural-looking gestures but face challenges in ensuring content matching between speech and generated gestures [Yoon et al 2022]. To address this issue, some neural systems aim to explicitly model both rhythm and semantics from the perspective of model structure [Ao et al 2022;Kucherenko et al 2021;Liu et al 2022a] or training supervision strategy ]. Furthermore, hybrid systems, such as the combination of deep features and motion graphs [Zhou et al 2022], have been proposed to harness the advantages of different approaches.…”

“…This raw motion representation, however, often contains redundant information. Following recent successful systems [Ao et al 2022;Dhariwal et al 2020;Rombach et al 2022], we learn a compact motion representation using VQ-VAE [van den Oord et al 2017] to ensure motion quality and diversity.…”

“…Following the convention of recent gesture generation research [Alexanderson et al 2020;Ao et al 2022;Ghorbani et al 2022], we evaluate the generated motions through a series of user studies. Quantitative evaluations are also conducted, with results provided in later sections as a reference.…”

“…A variety of gestures can correspond to the same speech while semantic gestures and their corresponding speech are usually not perfectly aligned temporally. Such ambiguities can cause an end-to-end deep learning system to learn a mean gesture motion and lose the semantic correspondence [Abzaliev et al 2022;Ao et al 2022;Kucherenko et al 2021]. To alleviate this problem, we learn a gesture-transcript joint embedding space using contrastive learning combined with a temporal aggregation mechanism.…”

Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation

“…In recent years, the compelling performance of deep neural networks has prompted datadriven approaches. Previous studies establish large-scale speech-gesture corpus to learn the mapping from speech audio to human skeletons in an end-to-end manner [4,5,25,27,30,34,39]. To attain more expressive results, Ginosar et al [16] and Yoon et al [41] propose GAN-based methods to guarantee realism by adversarial mechanism, where the discriminator is trained to distinguish real gestures from the synthetic ones while the generator's objective is to fool the discriminator.…”