Language2Pose: Natural Language Grounded Pose Forecasting

Abstract

Figure 1: Overview of our model which uses joint multimodal space of language and pose to generate an animation conditioned on the input sentence. AbstractGenerating animations from natural language sentences finds its applications in a a number of domains such as movie script visualization, virtual human animation and, robot motion planning. These sentences can describe different kinds of actions, speeds and direction of these actions, and possibly a target destination. The core modeling challenge in this language-to-pose application is how to map linguistic concepts to motion animations.In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language-to-Pose (or JL2P), which learns a joint embedding of language and pose. This joint embedding space is learned end-toend using a curriculum learning approach which emphasizes shorter and easier sequences first before moving to longer and harder ones. We evaluate our proposed model on a publicly available corpus of 3D pose data and humanannotated sentences. Both objective metrics and human judgment evaluation confirm that our proposed approach is able to generate more accurate animations and are deemed visually more representative by humans than other data driven approaches.

Dynamic Future Net

“…For all our experiments, we use CMU MoCap database 1 . CMU dataset is a high-quality dataset acquired using optical motion capture systems, containing 2605 trials in 6 categories and 23 subcategories.…”

“…In [35], the control signal becomes the 3d human pose predicted by neural nets as a reference for an agent to imitate. In [1], the authors co-embed the language and corresponding motions to a share manifold, ignoring the fact that language-to-motion is a one-to-many mapping. Even with a specific control signal, like 2D human skeleton, one can still expect that there are different motions or different pose corresponding to the same control signal [29], essentially indicating the multi-modality nature of human motion dynamics.…”

Embodying Pre-Trained Word Embeddings Through Robot Actions

“…This characteristic provides an ability to manage "unseen words" that do not appear in the paired dataset. Some studies generated actions from descriptions represented by pre-trained word embeddings [2][3] [6][11] [16]. Zhong et al [11], Matthews et al [3], and Lunch et al [2], in particular, generated actions from commands including unseen words.…”

Action2Motion

“…Pioneer efforts such as [1,18,24] mainly resort to encoder-decoder RNN architecture for languageto-pose translation. The work of [2] learns a joint embedding space between sentences and human pose sequences. More recently, [26] applies more sophisticated neural translation network equipped with GANs for text-to-sign prediction.…”