Semantic Exploration from Language Abstractions and Pretrained Representations

Abstract

Continuous first-person 3D environments pose unique exploration challenges to reinforcement learning (RL) agents, because of their high-dimensional state and action spaces. These challenges can be ameliorated by using semantically meaningful state abstractions to define novelty for exploration. We propose that learned representations shaped by natural language provide exactly this form of abstraction. In particular, we show that vision-language representations, when pretrained on image captioning datasets sampled from the internet, can drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by comparing the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach in two very different task domains-one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world-as well as two popular deep RL algorithms: Impala and R2D2. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments. * Equal contribution Preprint. Under review.

Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling

“…One drawback of DECKARD, along with many other LLMassisted RL methods, is that it requires an environment already be grounded in language. Some preliminary methods for generating state descriptions from images are used by Tam et al (2022), but this remains an open area of research.…”

“…In this work, we focus on using LLMs for exploration rather than directly generating action plans. Tam et al (2022) and Mu et al (2022) recently demonstrated that language is a meaningful state abstraction when used for exploration. Additionally, Tam et al (2022) ex-periment with using LLM latent representations of state descriptions for novelty exploration, relying on pretrained LLM encodings to detect novel textual states.…”

“…Tam et al (2022) and Mu et al (2022) recently demonstrated that language is a meaningful state abstraction when used for exploration. Additionally, Tam et al (2022) ex-periment with using LLM latent representations of state descriptions for novelty exploration, relying on pretrained LLM encodings to detect novel textual states. To the best of our knowledge, we are the first to apply language-assisted decision-making to exploration by using LLMs to predict and verify environment dynamics through experience.…”

Likelihood-Based Diverse Sampling for Trajectory Forecasting

“…Baselines. We compare against CLIP (Radford et al, 2021), a state-of-art vision-language representation that has seen wide adoption in various robotics tasks (Shridhar et al, 2022a;Cui et al, 2022;Khandelwal et al, 2022;Tam et al, 2022); as LIV is trained using the CLIP architecture and initialized with CLIP weights, this is the closest comparison. We also compare against R3M (Nair et al, 2022b) and VIP (Ma et al, 2022b), two state-of-art pre-trained visual representations.…”

Name that state: How language affects human reinforcement learning

“…One cognitive function that exhibits many of the properties thought to be important for state representation is language (Tam et al, 2022). Language representations are inherently low-dimensional and flexible at different levels of abstraction (Piantadosi, Tenenbaum, & Goodman, 2016;Antonello, Turek, Vo, & Huth, 2021).…”