GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

Tenglong, Ao,; Zhang, Z Y; Liu, Libin

doi:10.1145/3592097

Cited by 64 publications

(7 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, DALLE [RDN*22] enables users to describe an image in terms of content and style. Similar applications can be observed in the computer animation field such as the Human Motion Diffusion Model (MDM) [TRG*23], Speech Gesture Diffusion [AZL23], etc.…”

Section: Introductionmentioning

confidence: 78%

Cinematographic Camera Diffusion Model

Jiang,

Wang,

Christie

et al. 2024

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

Designing effective camera trajectories in virtual 3D environments is a challenging task even for experienced animators. Despite an elaborate film grammar, forged through years of experience, that enables the specification of camera motions through cinematographic properties (framing, shots sizes, angles, motions), there are endless possibilities in deciding how to place and move cameras with characters. Dealing with these possibilities is part of the complexity of the problem. While numerous techniques have been proposed in the literature (optimization‐based solving, encoding of empirical rules, learning from real examples,…), the results either lack variety or ease of control.In this paper, we propose a cinematographic camera diffusion model using a transformer‐based architecture to handle temporality and exploit the stochasticity of diffusion models to generate diverse and qualitative trajectories conditioned by high‐level textual descriptions. We extend the work by integrating keyframing constraints and the ability to blend naturally between motions using latent interpolation, in a way to augment the degree of control of the designers. We demonstrate the strengths of this text‐to‐camera motion approach through qualitative and quantitative experiments and gather feedback from professional artists. The code and data are available at https://github.com/jianghd1996/Camera-control.

show abstract

Section: Introductionmentioning

confidence: 78%

Cinematographic Camera Diffusion Model

Jiang,

Wang,

Christie

et al. 2024

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

show abstract

“…Similarly, [8] conditions on CLIP latents, but combines latent space based and diffusion based motion generation. Most similar to our work is [3], which learns a gesture-text joint embedding using contrastive learning and a CLIP based style encoding module in a diffusion based gesture synthesis model.…”

Section: Using Language Based Pre-training Approaches In Motion Gener...mentioning

confidence: 99%

“…Diffusion models [15,34,35] have emerged as a notable and contemporary probabilistic generative modelling methodology. These models have shown promise in capturing complex data distributions and have gained attention in various fields, including gesture generation [2,3,30,45]. Inspired by these works our system uses Denoising Diffusion Probabilistic Modelling (DDPM) [15] formulation with self-supervised representations to synthesise gestures conditioned on the input audio.…”

Section: Related Work 21 Co-speech Gesture Generationmentioning

confidence: 99%

Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation

Deichler,

Mehta,

Alexanderson

et al. 2023

International Conference on Multimodal Interaction

View full text Add to dashboard Cite

This paper describes a system developed for the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2023. Our solution builds on an existing diffusion-based motion synthesis model. We propose a contrastive speech and motion pretraining (CSMP) module, which learns a joint embedding for speech and gesture with the aim to learn a semantic coupling between these modalities. The output of the CSMP module is used as a conditioning signal in the diffusion-based gesture synthesis model in order to achieve semantically-aware co-speech gesture generation. Our entry achieved highest human-likeness and highest speech appropriateness rating among the submitted entries. This indicates that our system is a promising approach to achieve human-like co-speech gestures in agents that carry semantic meaning.

show abstract

“…CLIP [11] trained language and image information jointly by minimizing the distance between corresponding image and text embeddings, achieving zero-shot performance on downstream tasks. Due to the impressive performance of visionlanguage models [39], our research focuses on improving the alignment between images and semantic information using these models. The objective is to provide diffusion models with more accurate guidance signals.…”

Section: Vision and Language (Vl) Modelsmentioning

confidence: 99%

GDUI: Guided Diffusion Model for Unlabeled Images

Xie,

Zhao

2024

Algorithms

View full text Add to dashboard Cite

The diffusion model has made progress in the field of image synthesis, especially in the area of conditional image synthesis. However, this improvement is highly dependent on large annotated datasets. To tackle this challenge, we present the Guided Diffusion model for Unlabeled Images (GDUI) framework in this article. It utilizes the inherent feature similarity and semantic differences in the data, as well as the downstream transferability of Contrastive Language-Image Pretraining (CLIP), to guide the diffusion model in generating high-quality images. We design two semantic-aware algorithms, namely, the pseudo-label-matching algorithm and label-matching refinement algorithm, to match the clustering results with the true semantic information and provide more accurate guidance for the diffusion model. First, GDUI encodes the image into a semantically meaningful latent vector through clustering. Then, pseudo-label matching is used to complete the matching of the true semantic information of the image. Finally, the label-matching refinement algorithm is used to adjust the irrelevant semantic information in the data, thereby improving the quality of the guided diffusion model image generation. Our experiments on labeled datasets show that GDUI outperforms diffusion models without any guidance and significantly reduces the gap between it and models guided by ground-truth labels.

show abstract

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

Cited by 64 publications

References 57 publications

Cinematographic Camera Diffusion Model

Cinematographic Camera Diffusion Model

Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation

GDUI: Guided Diffusion Model for Unlabeled Images

Contact Info

Product

Resources

About