Caption Generation of Robot Behaviors based on Unsupervised Learning of Action Segments

Abstract

Bridging robot action sequences and their natural language captions is an important task to increase explainability of human assisting robots in their recently evolving field. In this paper, we propose a system for generating natural language captions that describe behaviors of human assisting robots. The system describes robot actions by using robot observations; histories from actuator systems and cameras, toward end-to-end bridging between robot actions and natural language captions. Two reasons make it challenging to apply existing sequence-tosequence models to this mapping: 1) it is hard to prepare a large-scale dataset for any kinds of robots and their environment, and 2) there is a gap between the number of samples obtained from robot action observations and generated word sequences of captions. We introduced unsupervised segmentation based on K-means clustering to unify typical robot observation patterns into a class. This method makes it possible for the network to learn the relationship from a small amount of data. Moreover, we utilized a chunking method based on byte-pair encoding (BPE) to fill in the gap between the number of samples of robot action observations and words in a caption. We also applied an attention mechanism to the segmentation task. Experimental results show that the proposed model based on unsupervised learning can generate better descriptions than other methods. We also show that the attention mechanism did not work well in our low-resource setting.

Do as I Demand, Not as I Say: A Dataset for Developing a Reflective Life-Support Robot

“…One possible solution is transferring the data collected in a simulated world to the real world (sim2real); however, transferring the knowledge acquired in simulated worlds remains a challenge [28]. Effective feature extraction must be investigated so that systems can work in actual situations [29].…”

“…In this study, we face a difficulty to collect a large amount of data because we use images that assume a robot's firstperson viewpoints in certain environments. Abstracting the dataset is critical for effectively using such a small amount of data as training data [29]. Feature extraction methods, with pre-trained models trained on large-scale data, have been widely used in recent years; however, more focused information is required to understand visual situations.…”

“…We also can use recent pre-trained models as feature extractors from both the utterance and the image [8], [33] to utilize multimodal features. However, we need to abstract information that can be extracted from the multimodal input when the model is trained with a small amount of training data for selecting robot action categories [29]. We investigated whether the classification accuracy could be improved by a classifier that utilizes our descriptive features such as objects or user poses.…”