Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents

Bhattacharya, Uttaran; Rewkowski, Nicholas; Banerjee, Abhishek; Guhan, Pooja; Bera, Aniket; Manocha, Dinesh

doi:10.1109/vr50410.2021.00037

Cited by 88 publications

(31 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mapping language to 3D human motions in a non-deterministic fashion is relatively new. Here we compare our method to four state-of-the-art methods: Seq2Seq [26], Language2Pose [2], Text2Gesture [5] and Hier [11]. As with all existing methods, they are unfortunately deterministic methods.…”

Section: Evaluation Of Text-to-motion Generationmentioning

confidence: 99%

“…Real motions 0.511 ±.003 0.703 ±.003 0.797 ±.002 0.002 ±.000 2.974 ±.008 9.503 ±.065 -Seq2Seq [26] 0.180 ±.002 0.300 ±.002 0.396 ±.002 11.75 ±.035 5.529 ±.007 6.223 ±.061 -Language2Pose [2] 0.246 ±.002 0.387 ±.002 0.486 ±.002 11.02 ±.046 5.296 ±.008 7.676 ±.058 -Text2Gesture [5] 0.165 ±.001 0.267 ±.002 0.345 ±.002 5.012 ±.030 6.030 ±.008 6.409 ±.071 -Hier [11] 0.301 ±.002 0.425 ±.002 0.552 ±.004 6.532 ±.024 5.012 ±.018 8.332 ±.042 -MoCoGAN [44] 0.037 ±.000 0.072 ±.001 0.106 ±.001 94.41 ±.021 9.643 ±.006 0.462 ±.008 0.019 ±.000 Dance2Music [24] 0.033 ±.000 0.065 ±.001 0.097 ±.001 66.98 ±.016 8.116 ±.006 0.725 ±.011 0.043 ±.001 Ours baseline(T) 0.351 ±.003 0.521 ±.003 0.627 ±.003 1.669…”

Section: Human Ml3dmentioning

confidence: 99%

“…Real motions 0.424 ±.005 0.649 ±.006 0.779 ±.006 0.031 ±.004 2.788 ±.012 11.08 ±.097 -Seq2Seq [26] 0.103 ±.003 0.178 ±.005 0.241 ±.006 24.86 ±.348 7.960 ±.031 6.744 ±.106 -Language2Pose [2] 0.221 ±.005 0.373 ±.004 0.483 ±.005 6.545 ±.072 5.147 ±.030 9.073 ±.100 -Text2Gesture [5] 0.156 ±.004 0.255 ±.004 0.338 ±.005 12.12 ±.183 6.964 ±.029 9.334 ±.079 -Hier [11] 0.255 ±.006 0.432 ±.007 0.531 ±.007 5.203 ±.107 4.986 ±.027 9.563 ±.072 -MoCoGAN [44] 0.022 ±.002 0.042 ±.003 0.063 ±.003 82.69 ±.242 10.47 ±.012 3.091 ±.043 0.250 ±.009 Dance2Music [24] 0.031 ±.002 0.058 ±.002 0.086 ±.003 115.4 ±.240 10.40 ±.016 0.241 ±.004 0.062 ±.002 Ours baseline(T) 0.260 ±.005 0.426 ±.007 0.538 ±.008 4.628 ±.126 4.835 ±.076 12.16 ±.120 4.436 ±.106 Ours baseline 0.251 ±.007 0.418 ±.008 0.535 ±.007 4.814 ±.145 4.682 ±.048 10.13 ±.117 4.486 ±.117 Ours 0.280 ±.005 0.463 ±.006 0.587 ±.005 3.599 ±.153 4.591 ±.026 9.473 ±.117 3.292 ±.081…”

Section: Kit-mlmentioning

confidence: 99%

“…suffers from severely low quality and diversity, as reflected by their low R precision and mutimodality score. Deterministic baselines such as Seq2Seq [26] and Text2Gesture [5] autoregressively regress human poses from textual input via vanilla sequence-to-sequence RNN and transformer respectively. However, such straightforward approaches find difficulty in maintaining textual semantics during generating human dynamics, which results in low motion-based text retrieval precision and high multimodal distance.…”

Section: Kit-mlmentioning

confidence: 99%

“…We extend their implementation with curriculum learning to enable the motion generation with variable lengths. Proper modifications are also made to the official implementations of Text2Gesture [5] and Language2Pose [2], to fit in our scenario such as kinematic structure. To adapt MoCoGAN [44] and Dance2Music [20] in our application, we re-use their source code and replace the categorical condition in MoCoGAN and audio signals in Dance2Music with our text features.…”

Section: Baseline Implementationmentioning

confidence: 99%

See 4 more Smart Citations

TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

Chen¹,

Xuo²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Inspired by the strong ties between vision and language, the two intimate human sensing and communication modalities, our paper aims to explore the generation of 3D human full-body motions from texts, as well as its reciprocal task, shorthanded for text2motion and motion2text, respectively. To tackle the existing challenges, especially to enable the generation of multiple distinct motions from the same text, and to avoid the undesirable production of trivial motionless pose sequences, we propose the use of motion token, a discrete and compact motion representation. This provides one level playing ground when considering both motions and text signals, as the motion and text tokens, respectively. Moreover, our motion2text module is integrated into the inverse alignment process of our text2motion training pipeline, where a significant deviation of synthesized text from the input text would be penalized by a large training loss; empirically this is shown to effectively improve performance. Finally, the mappings in-between the two modalities of motions and texts are facilitated by adapting the neural model for machine translation (NMT) to our context. This autoregressive modeling of the distribution over discrete motion tokens further enables non-deterministic production of pose sequences, of variable lengths, from an input text. Our approach is flexible, could be used for both text2motion and motion2text tasks. Empirical evaluations on two benchmark datasets demonstrate the superior performance of our approach on both tasks over a variety of state-of-the-art methods. Project page: https://ericguo5513.github.io/TM2T/

show abstract

Section: Evaluation Of Text-to-motion Generationmentioning

confidence: 99%

Section: Human Ml3dmentioning

confidence: 99%

Section: Kit-mlmentioning

confidence: 99%

Section: Kit-mlmentioning

confidence: 99%

Section: Baseline Implementationmentioning

confidence: 99%

See 3 more Smart Citations

TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

Chen¹,

Xuo²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Music conditioned 2D hand gesture dance generation with HGS

Zhou,

Liu,

2023

Computer Animation & Virtual

View full text Add to dashboard Cite

In recent years, the short video industry is booming. However, there are still many difficulties in the action generation of virtual characters. We observed that on the short video social platform, “hand gesture dance” is a very popular short video form. However, its development is limited by the professionalism of choreography. In order to solve these problems, we propose an intelligent choreography framework, which can generate new gesture sequences for unseen audio based on pairing data in the database. Our framework adopts multimodal method and obtains excellent results. In additional, we collected and produced the first and largest pair labeled hand gesture dance data set. Various experiments showed that our results not only generate smooth and rich action sequences, but also collect some semantic information contained in the audio.

show abstract

TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

Guo

Zuo

Wang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents

Cited by 88 publications

References 52 publications

TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

Music conditioned 2D hand gesture dance generation with HGS

TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts

Contact Info

Product

Resources

About