Imitating Interactive Intelligence

Abstract

A common vision from science fiction is that robots will one day inhabit our physical spaces, sense the world as we do, assist our physical labours, and communicate with us through natural language. Here we study how to design artificial agents that can interact naturally with humans using the simplification of a virtual environment. This setting nevertheless integrates a number of the central challenges of artificial intelligence (AI) research: complex visual perception and goal-directed physical control, grounded language comprehension and production, and multi-agent social interaction. To build agents that can robustly interact with humans, we would ideally train them while they interact with humans. However, this is presently impractical. Therefore, we approximate the role of the human with another learned agent, and use ideas from inverse reinforcement learning to reduce the disparities between human-human and agent-agent interactive behaviour. Rigorously evaluating our agents poses a great challenge, so we develop a variety of behavioural tests, including evaluation by humans who watch videos of agents or interact directly with them. These evaluations convincingly demonstrate that interactive training and auxiliary losses improve agent behaviour beyond what is achieved by supervised learning of actions alone. Further, we demonstrate that agent capabilities generalise beyond literal experiences in the dataset. Finally, we train evaluation models whose ratings of agents agree well with human judgement, thus permitting the evaluation of new agent models without additional effort. Taken together, our results in this virtual environment provide evidence that large-scale human behavioural imitation is a promising tool to create intelligent, interactive agents, and the challenge of reliably evaluating such agents is possible to surmount. See videos for an overview of the manuscript, training time-lapse, and human-agent interactions.

Collaborating with Humans without Human Data

“…We also focus on one domain in order to compare with prior work which has argued that human-in-the-loop training is necessary. Consequently, the resulting agents are only designed to adaptively collaborate on a single task, and not to infer human preferences in general [1,32,58]. Moreover, if a task's reward function is poorly aligned with how humans approach the task, our method may well produce subpar partners, as would any method without access to human data.…”

SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition

“…Our results are based on three procedurally generated video datasets of multi-object scenes. In increasing order of difficulty, they are: Objects Room 9 [42], CATER (moving camera) [43], and Playroom [44]. These were chosen to meet a number of criteria: we wanted at least 9-10 objects per scene (there could be fewer in view, or as many as 25 in the case of Playroom).…”

“…The Playroom is a Unity-based environment for object-centric tasks [67,44], originally released as pre-packaged Docker containers with an Apache 2.0 license. We used an arbitrary behaviour policy (trained by demonstrations) to generate video sequences from the environment (one per episode).…”

What underlies rapid learning and systematic generalization in humans

“…In interpreting the effect of goal position, we note that, throughout the test phase, 25% of the puzzles used the same goal cell as the puzzles during the tutorial and practice phases, whereas when the goal position was changed, the goal cell would be any one of 16 possible cells with 1 32 probability or 64 cells with 1 256 probability. Thus, the persistent effect of goal position might result from a justifiable bias in attention toward the most common goal cell location, producing a small cost when attention must be deployed to a less likely position.…”

“…A further important extension will be to allow the simultaneous use of language to receive and generate instructions and explanations as partially implemented in [1]. This model could learn both by independently interacting with the environment as in traditional reinforcement learning, but also from a structured curriculum of mathematical problemsolving tasks requiring following instructions and receiving and producing explanations.…”