Natural Language Communication with Robots

Abstract

We propose a framework for devising empirically testable algorithms for bridging the communication gap between humans and robots. We instantiate our framework in the context of a problem setting in which humans give instructions to robots using unrestricted natural language commands, with instruction sequences being subservient to building complex goal configurations in a blocks world. We show how one can collect meaningful training data and we propose three neural architectures for interpreting contextually grounded natural language commands. The proposed architectures allow us to correctly understand/ground the blocks that the robot should move when instructed by a human who uses unrestricted language. The architectures have more difficulty in correctly understanding/grounding the spatial relations required to place blocks correctly, especially when the blocks are not easily identifiable.

Communication with Robots using Multilayer Recurrent Networks

“…This is an improvement over both rule based benchmark with 1.54 and the best model reported by Bisk et al (2016b), who had 0.98. The median distance is 0.04 which is much better than their comparable End-To-End model with median distance 0.53.…”

“…In this paper, we propose several models solving this task and report improvement compared to the previous work by Bisk et al (2016b).…”

“…The first approach using neural networks is proposed by Bisk et al (2016b), who describe and compare several neural models for understanding natural language commands. Their dataset (Bisk et al, 2016a) contains simulated world with square blocks and actions descriptions in English (see Figure 1).…”

Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

“…Environment We use the environment of Bisk et al (2016). The original task required predicting the source and target positions for a single block given an instruction.…”

“…This approach offers multiple benefits, such as not requiring intermediate representations, planning procedures, or training multiple models. Figure 1 illustrates the problem in the Blocks environment (Bisk et al, 2016). The agent observes the environment as an RGB image using a camera sensor.…”