Embodying Pre-Trained Word Embeddings Through Robot Actions

Abstract

We propose a promising neural network model with which to acquire a grounded representation of robot actions and the linguistic descriptions thereof. Properly responding to various linguistic expressions, including polysemous words, is an important ability for robots that interact with people via linguistic dialogue. Previous studies have shown that robots can use words that are not included in the action-description paired datasets by using pre-trained word embeddings. However, the word embeddings trained under the distributional hypothesis are not grounded, as they are derived purely from a text corpus. In this paper, we transform the pre-trained word embeddings to embodied ones by using the robot's sensory-motor experiences. We extend a bidirectional translation model for actions and descriptions by incorporating non-linear layers that retrofit the word embeddings. By training the retrofit layer and the bidirectional translation model alternately, our proposed model is able to transform the pre-trained word embeddings to adapt to a paired action-description dataset. Our results demonstrate that the embeddings of synonyms form a semantic cluster by reflecting the experiences (actions and environments) of a robot. These embeddings allow the robot to properly generate actions from unseen words that are not paired with actions in a dataset.

Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data

“…During the conversion between symbolic actions/states and texts in a 3D simulator environment [5], pre-training on large non-paired data has shown an improved performance in zero-shot settings. For description-from/to-action translation, [2] realized appropriate action generation that accepts unseen words not included in the dataset. They successfully retrofitted the pretrained word embeddings into multimodal representations incorporating the action modality.…”

“…The present study also adopts this method, following other studies of bidirectional translation between descriptions and actions [1] [2]. A bi-modal autoencoder model has been proposed for the acquisition of multimodal representations of vision and language [11].…”

“…In this study, we use a retrofitted paired recurrent autoencoder (rPRAE [2]) to perform bidirectional translation between descriptions and actions. An overview of the model is presented in Fig.…”

“…To achieve this ability, the robot needs to associate real-world objects with linguistic expressions, and complete paired data are generally required to acquire that relationship. In particular, the machine-learning-based approaches [1] [2] require the preparation of a large paired dataset of descriptions and actions. However, collecting and captioning behavioral data is costly, making it difficult to collect large amounts of paired data.…”

“…Several joint learning methods for language and motion have been proposed to address these issues. Using contextindependent word embeddings derived from a large corpus, [2] and [3] dealt with unseen words that were not included in the training data. In other studies, stepwise learning methods were proposed to integrate multiple models from different domains [4] [5].…”