Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment

Abstract

First-person object-interaction tasks in high-fidelity, 3D, simulated environments such as the AI2Thor virtual home-environment pose significant sample-efficiency challenges for reinforcement learning (RL) agents learning from sparse task rewards. To alleviate these challenges, prior work has provided extensive supervision via a combination of reward-shaping, ground-truth object-information, and expert demonstrations. In this work, we show that one can learn object-interaction tasks from scratch without supervision by learning an attentive object-model as an auxiliary task during task learning with an object-centric relational RL agent. Our key insight is that learning an object-model that incorporates object-attention into forward prediction provides a dense learning signal for unsupervised representation learning of both objects and their relationships. This, in turn, enables faster policy learning for an object-centric relational RL agent. We demonstrate our agent by introducing a set of challenging object-interaction tasks in the AI2Thor environment where learning with our attentive object-model is key to strong performance. Specifically, we compare our agent and relational RL agents with alternative auxiliary tasks to a relational RL agent equipped with ground-truth object-information, and show that learning with our object-model best closes the performance gap in terms of both learning speed and maximum success rate. Additionally, we find that incorporating object-attention into an object-model's forward predictions is key to learning representations which capture object-category and object-state.

Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos

“…Our work lies along this line of research. Other approaches have considered using contrastive losses (Kipf et al, 2019;Carvalho et al, 2020) while some works have explored unsupervised object-centric learning in 3D scenes (Chen et al, 2021;Crawford & Pineau, 2020;Stelzner et al, 2021;Du et al, 2021b;Kabra et al, 2021). While dealing with 3D scenes would be an important future direction for our work, it is orthogonal to our current focus.…”

Learning Compositional Tasks from Language Instructions

“…vision-language navigation (Anderson et al 2018) and object interaction (Shridhar et al 2020;Carvalho et al 2020;Corona et al 2020;Blukis et al 2022;Min et al 2021) in embodied settings. In particular, the AL-FRED task (Shridhar et al 2020) studies agents that exploit detailed natural language instructions to generalize to novel instructions in novel environments at test time.…”

Learning Parameterized Task Structure for Generalization to Unseen Entities

“…Real world tasks are also compositional (Carvalho et al 2020;Loula, Baroni, and Lake 2018;Andreas, Klein, and Levine 2017;Oh et al 2017). Compositional tasks are often made of different "components" that can recombined to form new tasks.…”