Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders

Li, Jing; Kang, Di; Pei, Wenjie; Zhe, Xuefei; Zhang, Ying; He, Zhenyu; Bao, Linchao

doi:10.1109/iccv48922.2021.01110

Cited by 78 publications

(26 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Probabilistic frameworks were also used in the recent gesture generation works to handle the gesture ambiguities [1], [2], [25], [20], [6]. This type of methods learns a latent space generative model and could generate multiple gesture motions from the same speech input using conditional sampling from the latent space during inference.…”

Section: Related Work a Co-speech Gesture Generationmentioning

confidence: 99%

“…Adding adversarial training schemes [13], [31] improved the results, but at the cost of a more elaborated training process that requires careful tuning. More recent works explored the probabilistic framework [2], [20] to address the non-deterministic nature of gesture synthesis and sample new gestures during inferences. This line of work builds a latent space model (e.g., normalizing flows) from gesture motions to learn its conditional probability distributions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Co-Speech Gesture Synthesis using Discrete Gesture Token Learning

Lu¹,

Yoon²,

Feng³

2023

Preprint

View full text Add to dashboard Cite

Synthesizing realistic co-speech gestures is an important and yet unsolved problem for creating believable motions that can drive a humanoid robot to interact and communicate with human users. Such capability will improve the impressions of the robots by human users and will find applications in education, training, and medical services. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. The deterministic regression methods can not resolve the conflicting samples and may produce over-smoothed or damped motions. We proposed a two-stage model to address this uncertainty issue in gesture synthesis by modeling the gesture segments as discrete latent codes. Our method utilizes RQ-VAE in the first stage to learn a discrete codebook consisting of gesture tokens from training data. In the second stage, a two-level autoregressive transformer model is used to learn the prior distribution of residual codes conditioned on input speech context. Since the inference is formulated as token sampling, multiple gesture sequences could be generated given the same speech input using top-k sampling. The quantitative results and the user study showed the proposed method outperforms the previous methods and is able to generate realistic and diverse gesture motions.

show abstract

Section: Related Work a Co-speech Gesture Generationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Co-Speech Gesture Synthesis using Discrete Gesture Token Learning

Lu¹,

Yoon²,

Feng³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Traum et al [39] used LSTMs to synthesise head movements from speech. Despite the use of RNNs in previous models, Li et al [21] shown that convolutional layers are better in the movements generation task, as it prevents the error accumulation, which is specific to RNN.…”

Section: Data-driven Approachesmentioning

confidence: 99%

“…Indeed, we did not use LSTMs but 1D convolution layers. This choice is motivated by the fact that Li et al [21] have shown that convolutional layers are better in the movements generation task, as it prevents the error accumulation, which is specific to RNN. Moreover, our outputs are not as Wu et al [42] who use only the 3D coordinates of the upper body movements.…”

Section: Data Normalisationmentioning

confidence: 99%

Automatic facial expressions, gaze direction and head movements generation of a virtual agent

Delbosc

Ochs

Ayache

2022

International Conference on Multimodal Interaction

View full text Add to dashboard Cite

In this article, we present two models to jointly and automatically generate the head, facial and gaze movements of a virtual agent from acoustic speech features. Two architectures are explored: a Generative Adversarial Network and an Adversarial Encoder-Decoder. Head movements and gaze orientation are generated as 3D coordinates, while facial expressions are generated using action units based on the facial action coding system. A large corpus of almost 4 hours of videos, involving 89 different speakers is used to train our models. We extract the speech and visual features automatically from these videos using existing tools. The evaluation of these models is conducted objectively with measures such as density evaluation and a visualisation from PCA reduction, as well as subjectively through a users perceptive study. Our proposed methodology shows that on 15 seconds sequences, encoder-decoder architecture drastically improves the perception of generated behaviours in two criteria: the coordination with speech and the naturalness. Our code can be found in : https://github.com/aldelb/nonverbal-behaviours-generation.

show abstract

“…Recently a new paradigm demonstrating the use of multi-modal data has emerged. It has been shown of tremendous success for generating poses based on audio data in dance sequences [18,21,32], for speech gesture generation [11,20], and for learning new multi-modal features from visual and sound data that can be used with a variety of classic vision or audio tasks [2].…”

Section: Pose Estimation With Audiomentioning

confidence: 99%

AIMusicGuru: Music Assisted Human Pose Correction

Shrestha¹,

Fermüller²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pose Estimation techniques rely on visual cues available through observations represented in the form of pixels. But the performance is bounded by the frame rate of the video and struggles from motion blur, occlusions, and temporal coherence. This issue is magnified when people are interacting with objects and instruments, for example playing the violin. Standard approaches for postprocessing use interpolation and smoothing functions to filter noise and fill gaps, but they cannot model highly non-linear motion. We present a method that leverages our understanding of the high degree of a causal relationship between the sound produced and the motion that produces them. We use the audio signature to refine and predict accurate human body pose motion models. We propose MAPnet (Music Assisted Pose network) for generating a fine grain motion model from sparse input pose sequences but continuous audio. To accelerate further research in this domain, we also open-source MAPdat, a new multi-modal dataset of 3D violin playing motion with music. We perform a comparison of different standard machine learning models and perform analysis on input modalities, sampling techniques, and audio and motion features. Experiments on MAPdat suggest multi-modal approaches like ours as a promising direction for tasks previously approached with visual methods only. Our results show both qualitatively and quantitatively how audio can be combined with visual observation to help improve any pose estimation methods.

show abstract

Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders

Cited by 78 publications

References 23 publications

Co-Speech Gesture Synthesis using Discrete Gesture Token Learning

Co-Speech Gesture Synthesis using Discrete Gesture Token Learning

Automatic facial expressions, gaze direction and head movements generation of a virtual agent

AIMusicGuru: Music Assisted Human Pose Correction

Contact Info

Product

Resources

About