World Discovery Models

Azar, Mohammad Gheshlaghi; Piot, Bilal; Pires, Bernardo Avila; Grill, Jean-Bastien; Altché, Florent; Munos, Rémi

doi:10.48550/arxiv.1902.07685

Cited by 4 publications

(4 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In RL, this intuition is transposed by taking the current state and action to predict the next state representation; the resulting prediction error is then turned into the intrinsic reward signal. Approaches mostly differ in learning the state representation: Finally, Achiam and Sastry (2017); Azar et al (2019) computes the intrinsic reward across multiple timesteps predictions to better estimate information gain. Yet, those intrinsic rewards based on prediction errors may attract the agent into irrelevant yet unpredictable transitions.…”

Section: Related Workmentioning

confidence: 99%

Don’t Do What Doesn’t Matter: Intrinsic Motivation with Action Usefulness

Seurin

Strub²,

Preux

et al. 2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Sparse rewards are double-edged training signals in reinforcement learning: easy to design but hard to optimize. Intrinsic motivation guidances have thus been developed toward alleviating the resulting exploration problem. They usually incentivize agents to look for new states through novelty signals. Yet, such methods encourage exhaustive exploration of the state space rather than focusing on the environment's salient interaction opportunities. We propose a new exploration method, called Don't Do What Doesn't Matter (DoWhaM), shifting the emphasis from state novelty to state with relevant actions. While most actions consistently change the state when used, e.g. moving the agent, some actions are only effective in specific states, e.g., opening a door, grabbing an object. DoWhaM detects and rewards actions that seldom affect the environment. We evaluate DoWhaM on the procedurally-generated environment MiniGrid against state-of-the-art methods. Experiments consistently show that DoWhaM greatly reduces sample complexity, installing the new state-of-the-art in MiniGrid.

show abstract

Section: Related Workmentioning

confidence: 99%

Don’t Do What Doesn’t Matter: Intrinsic Motivation with Action Usefulness

Seurin

Strub²,

Preux

et al. 2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…Information gain can be estimated by planning (Sun et al, 2011) or from past environment interaction (Schmidhuber, 1991). State representations lead to agents that disambiguate unobserved environment states, for example by opening doors to see objects behind them, such as in active inference (Da Costa et al, 2020), INDIGO (Azar et al, 2019), and DVBF-LM (Mirchev et al, 2018). Model parameters lead to agents that discover the rules of their environment, such as in active inference (Friston et al, 2015), VIME (Houthooft et al, 2016), MAX (Shyam et al, 2018), and Plan2Explore (Sekar et al, 2020).…”

Section: Information Gainmentioning

confidence: 99%

Action and Perception as Divergence Minimization

Hafner¹,

Ortéga²,

Ba³

et al. 2020

Preprint

View full text Add to dashboard Cite

We introduce a unified objective for action and perception of intelligent agents. Extending representation learning and control, we minimize the joint divergence between the combined system of agent and environment and a target distribution. Intuitively, such agents use perception to align their beliefs with the world, and use actions to align the world with their beliefs. Minimizing the joint divergence to an expressive target maximizes the mutual information between the agent's representations and inputs, thus inferring representations that are informative of past inputs and exploring future inputs that are informative of the representations. This lets us explain intrinsic objectives, such as representation learning, information gain, empowerment, and skill discovery from minimal assumptions. Moreover, interpreting the target distribution as a latent variable model suggests powerful world models as a path toward highly adaptive agents that seek large niches in their environments, rendering task rewards optional. The framework provides a common

show abstract

“…Neural population decoding. Probing an agent with a QA decoder can be viewed as a variant of neural population decoding, used as an analysis tool in neuroscience (Georgopoulos et al, 1986;Bialek et al, 1991;Salinas & Abbott, 1994) and more recently in deep learning (Guo et al, 2018;Gregor et al, 2019;Azar et al, 2019;Alain & Bengio, 2016;Conneau et al, 2018;Tenney et al, 2019). The idea is to test whether specific information is encoded in a learned representation, by feeding the representation as input to a probe network, generally a classifier trained to extract the desired information.…”

Section: Background and Related Workmentioning

confidence: 99%

Probing Emergent Semantics in Predictive Agents via Question Answering

Das,

Carnevale,

Merzic

et al. 2020

Preprint

View full text Add to dashboard Cite

Recent work has shown how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments. We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling -actionconditional CPC (Guo et al., 2018) and Sim-Core (Gregor et al., 2019). After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent. The performance of different agents when probed this way reveals that they learn to encode factual, and seemingly compositional, information about objects, properties and spatial relations from their physical environment. Our approach is intuitive, i.e. humans can easily interpret responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach. By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

show abstract

World Discovery Models

Cited by 4 publications

References 40 publications

Don’t Do What Doesn’t Matter: Intrinsic Motivation with Action Usefulness

Don’t Do What Doesn’t Matter: Intrinsic Motivation with Action Usefulness

Action and Perception as Divergence Minimization

Probing Emergent Semantics in Predictive Agents via Question Answering

Contact Info

Product

Resources

About