UEA Digital Humans entry to the GENEA Challenge 2022

Windle, Jonathan; Greenwood, David A.; Taylor, Sarah L.

doi:10.1145/3536221.3558065

Cited by 7 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Selecting optimal representations for conditional input is a crucial research challenge in creating virtual human motions [18] [19]. For accurate reflection of gestures that match the auditory perception, prevalent research [7], [20]- [22] utilizes preprocessed audio features, such as MFCCs, log amplitude spectrogram, etc.…”

Section: A Condition Extraction Mechanismmentioning

confidence: 99%

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Zhang,

Wang,

Lyu

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the expressiveness of resulting gestures and limit their applicability. To address these challenges, we present Persona-Gestor, a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures solely relying on raw speech audio. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture. The fuzzy feature extractor harnesses a fuzzy inference strategy that automatically infers implicit, continuous fuzzy features. These fuzzy features, represented as a unified latent feature, are fed into the AdaLN transformer. The AdaLN transformer introduces a conditional mechanism that applies a uniform function across all tokens, thereby effectively modeling the correlation between the fuzzy features and the gesture sequence. This module ensures a high level of gesture-speech synchronization while preserving naturalness. Finally, we employ the diffusion model to train and infer various gestures. Extensive subjective and objective evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's superior performance to the current state-of-theart approaches. Persona-Gestor improves the system's usability and generalization capabilities, setting a new benchmark in speech-driven gesture synthesis and broadening the horizon for virtual human technology. Supplementary videos and code can be accessed at https://zf223669.github.io/Diffmotion-v2-website/.

show abstract

Section: A Condition Extraction Mechanismmentioning

confidence: 99%

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Zhang,

Wang,

Lyu

et al. 2024

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

Section: A Condition Extraction Mechanismmentioning

confidence: 99%

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Zhang,

Wang,

Lyu

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Speech-driven gesture generation is an emerging field within the domain of virtual human creation. The primary objective in this field is to attain authentic and personalized co-speech gestures while considering appropriate input conditions. However, a significant challenge lies in the difficulty of accurately determining the multitude of factors (such as acoustic, semantic, emotional, personality, and even subtle unknown features) inherent in these input conditions, which can be considered a complex class of fuzzy sets. Consequently, relying solely on explicit classification labels by manual annotation imposes limitations on the potential diversity of output states. To address these challenges, we introduce \textit{Persona-Gestor}, a novel approach integrating an automatic fuzzy feature inference mechanism with a probabilistic diffusion-based non-autoregressive transformer model. The fuzzy feature inference mechanism, embedded within a condition extractor, automatically extracts feature sets solely from raw speech audio data. These extracted features are subsequently employed as input conditions to facilitate the generation of personalized 3D full-body gestures. The condition extractor effectively leverages the WavLM large-scale pre-trained model to seamlessly capture local and global audio information into a unified latent representation associated with gestures. This all pertinent information eliminates the necessity for manual annotation labels, thereby streamlining multimodal processing. Furthermore, we employ adaptive layer normalization to enhance the modeling of the intricate relationship between speech and gestures. Finally, the learning and synthesis stages are facilitated through a diffusion process, leading to a wide range of gesture-generation outcomes. Extensive subjective and objective evaluations conducted on three high-quality co-speech gesture datasets (Trinity, ZEGGS, and BEAT) demonstrate our method's superior performance compared to recent approaches.</p>

show abstract

Llanimation: Llama Driven Gesture Animation

Windle,

Matthews,

Taylor

2024

Computer Graphics Forum

View full text Add to dashboard Cite

Co‐speech gesturing is an important modality in conversation, providing context and social cues. In character animation, appropriate and synchronised gestures add realism, and can make interactive agents more engaging. Historically, methods for automatically generating gestures were predominantly audio‐driven, exploiting the prosodic and speech‐related content that is encoded in the audio signal. In this paper we instead experiment with using Large‐Language Model (LLM) features for gesture generation that are extracted from text using Llama2. We compare against audio features, and explore combining the two modalities in both objective tests and a user study. Surprisingly, our results show that Llama2 features on their own perform significantly better than audio features and that including both modalities yields no significant difference to using Llama2 features in isolation. We demonstrate that the Llama2 based model can generate both beat and semantic gestures without any audio input, suggesting LLMs can provide rich encodings that are well suited for gesture generation.

show abstract

UEA Digital Humans entry to the GENEA Challenge 2022

Cited by 7 publications

References 15 publications

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Llanimation: Llama Driven Gesture Animation

Contact Info

Product

Resources

About