GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

Tenglong, Ao,; Zhang, Zeyi; Liu, Libin

doi:10.48550/arxiv.2303.14613

Cited by 5 publications

(12 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To compute this metric, we utilize the autoencoder pre-trained by BEAT (Liu et al 2022a). (Yoon et al 2019;Ginosar et al 2019;Yoon et al 2020;Li et al 2021a;Liu et al 2022a;Yi et al 2022;Ao, Zhang, and Liu 2023) in the term of FGD, SRGR and BeatAlign. All methods are trained on BEAT datasets.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…While substantial progress has been made in generating gestures synchronized to audio (Ginosar et al 2019;Qian et al 2021;Yazdian, Chen, and Lim 2022;Yang et al 2023d;Ao, Zhang, and Liu 2023;Yang et al 2023a), there has been limited exploration of emotive gesture generation that Figure 1: Performance comparison with limited modal during inference. The performance of existing multimodal methods is significantly hindered by the inadequate incorporation of multiple modalities during the inference stage.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control

Xu,

Zhang,

Yang

et al. 2024

AAAI

View full text Add to dashboard Cite

This study aims to improve the generation of 3D gestures by utilizing multimodal information from human speech. Previous studies have focused on incorporating additional modalities to enhance the quality of generated gestures. However, these methods perform poorly when certain modalities are missing during inference. To address this problem, we suggest using speech-derived multimodal priors to improve gesture generation. We introduce a novel method that separates priors from speech and employs multimodal priors as constraints for generating gestures. Our approach utilizes a chain-like modeling method to generate facial blendshapes, body movements, and hand gestures sequentially. Specifically, we incorporate rhythm cues derived from facial deformation and stylization prior based on speech emotions, into the process of generating gestures. By incorporating multimodal priors, our method improves the quality of generated gestures and eliminate the need for expensive setup preparation during inference. Extensive experiments and user studies confirm that our proposed approach achieves state-of-the-art performance.

show abstract

Section: Evaluation Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control

Xu,

Zhang,

Yang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…LDA [4] enables the system to generate style gestures with classifierfree guidance. Additionally, recent research has explored using textual prompts to generate stylized gestures [16]. Given that human emotions are more accurately represented on a continuous spectrum [32] [33] and emerge from a complex interplay of fuzzy factors, depending on discrete emotion labels can overly simplify the gesture generation process.…”

Section: A Condition Extraction Mechanismmentioning

confidence: 99%

“…For the Trinity dataset, we employed LDA [4] and Taming [15]. In addition to LDA and Taming, for the ZEGGS dataset, we also incorporated DiffuseStyleGesture (DSG) [2] and Ze-roEGGS [7] Furthermore, for the BEAT dataset, we utilized the same baseline models as in ZEGGS but replaced DSG with DSG+ [3] and introduced GestureDiffuCLIP (GDC) [16] as an additional baseline model.…”

Section: Subjective and Objective Evaluationmentioning

confidence: 99%

“…This methodological shift has led to substantial progress in the efficiency and flexibility of gesture-generation technologies. Key examples of such innovative efforts include Taming [15], Diffuse Style Gesture [2], Diffuse Style Gesture+ [3], GestureDiffuClip [16], and LDA [4]. Yet, these approaches encounter challenges with either insufficient or excessive correlation between gesture and speech, reducing the naturalness of the generated gestures.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Zhang,

Wang,

Lyu

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Speech-driven gesture generation is an emerging field within the domain of virtual human creation. The primary objective in this field is to attain authentic and personalized co-speech gestures while considering appropriate input conditions. However, a significant challenge lies in the difficulty of accurately determining the multitude of factors (such as acoustic, semantic, emotional, personality, and even subtle unknown features) inherent in these input conditions, which can be considered a complex class of fuzzy sets. Consequently, relying solely on explicit classification labels by manual annotation imposes limitations on the potential diversity of output states. To address these challenges, we introduce \textit{Persona-Gestor}, a novel approach integrating an automatic fuzzy feature inference mechanism with a probabilistic diffusion-based non-autoregressive transformer model. The fuzzy feature inference mechanism, embedded within a condition extractor, automatically extracts feature sets solely from raw speech audio data. These extracted features are subsequently employed as input conditions to facilitate the generation of personalized 3D full-body gestures. The condition extractor effectively leverages the WavLM large-scale pre-trained model to seamlessly capture local and global audio information into a unified latent representation associated with gestures. This all pertinent information eliminates the necessity for manual annotation labels, thereby streamlining multimodal processing. Furthermore, we employ adaptive layer normalization to enhance the modeling of the intricate relationship between speech and gestures. Finally, the learning and synthesis stages are facilitated through a diffusion process, leading to a wide range of gesture-generation outcomes. Extensive subjective and objective evaluations conducted on three high-quality co-speech gesture datasets (Trinity, ZEGGS, and BEAT) demonstrate our method's superior performance compared to recent approaches.</p>

show abstract

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Zhang,

Wang,

Lyu

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the expressiveness of resulting gestures and limit their applicability. To address these challenges, we present Persona-Gestor, a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures solely relying on raw speech audio. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture. The fuzzy feature extractor harnesses a fuzzy inference strategy that automatically infers implicit, continuous fuzzy features. These fuzzy features, represented as a unified latent feature, are fed into the AdaLN transformer. The AdaLN transformer introduces a conditional mechanism that applies a uniform function across all tokens, thereby effectively modeling the correlation between the fuzzy features and the gesture sequence. This module ensures a high level of gesture-speech synchronization while preserving naturalness. Finally, we employ the diffusion model to train and infer various gestures. Extensive subjective and objective evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's superior performance to the current state-of-theart approaches. Persona-Gestor improves the system's usability and generalization capabilities, setting a new benchmark in speech-driven gesture synthesis and broadening the horizon for virtual human technology. Supplementary videos and code can be accessed at https://zf223669.github.io/Diffmotion-v2-website/.

show abstract

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

Cited by 5 publications

References 54 publications

Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control

Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Contact Info

Product

Resources

About