Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks

Abstract

Linking human whole-body motion and natural language is of great interest for the generation of semantic representations of observed human behaviors as well as for the generation of robot behaviors based on natural language input. While there has been a large body of research in this area, most approaches that exist today require a symbolic representation of motions (e.g. in the form of motion primitives), which have to be defined a-priori or require complex segmentation algorithms. In contrast, recent advances in the field of neural networks and especially deep learning have demonstrated that sub-symbolic representations that can be learned end-to-end usually outperform more traditional approaches, for applications such as machine translation. In this paper we propose a generative model that learns a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks (RNNs) and sequence-to-sequence learning. Our approach does not require any segmentation or manual feature engineering and learns a distributed representation, which is shared for all motions and descriptions. We evaluate our approach on 2 846 human whole-body motions and 6 187 natural language descriptions thereof from the KIT Motion-Language Dataset. Our results clearly demonstrate the effectiveness of the proposed model: We show that our model generates a wide variety of realistic motions only from descriptions thereof in form of a single sentence. Conversely, our model is also capable of generating correct and detailed natural language descriptions from human motions.

Text2Action: Generative Adversarial Synthesis from Language to Action

“…In addition, it has been proven that a GAN has a unique solution, in which G captures the distribution of the real data and D does not distinguish the real data from the data generated from G [8]. Thanks to these features of GANs, our experiment also shows that GANs can generate more realistic action than the previous work [5].…”

“…The result shows that our generative model synthesizes the human action sequence that is more similar to the data. Although the network presented in [5] also generates the action as the ballet player with both arms open, it is shown that the action sequence synthesized by our network is more natural and similar to the data.…”

“…In order to see the difference between our proposed network and the network of [5], we have implemented the network presented in [5] based on the Tensorflow and trained it with our dataset. First, we compare generated actions when we give the sentence 'Woman dancing ballet with man', which is included in the training dataset, as an input to the each network.…”

“…There are datasets which provide human whole body motions and corresponding word or sentence annotations [2], [3]. In additions, there have been attempts for learning the mapping between language and human action [4], [5]. In [4], hidden Markov models (HMMs) [6] is used to encode motion primitives and to associate them with words.…”

“…In [4], hidden Markov models (HMMs) [6] is used to encode motion primitives and to associate them with words. [5] used a sequence to sequence (SEQ2SEQ) model [7] to learn the relationship between the natural language and the human actions.…”