Learning Latent Dynamics for Planning from Pixels

Hafner, Danijar; Lillicrap, Timothy P.; Fischer, Ian; Villegas, Ruben; Ha, David; Lee, Honglak; Davidson, James

doi:10.48550/arxiv.1811.04551

Cited by 100 publications

(221 citation statements)

References 0 publications

Supporting

Mentioning

220

Contrasting

Order By: Relevance

“…Rich image representations are useful for many downstream tasks. For example, VAEs are often used for latent representation learning in model-based reinforcement learning [8,10,11] by predicting possible future outcomes of the environment. VQ-VAE's latent variables are used as a means for video generation tasks [46].…”

Section: Related Workmentioning

confidence: 99%

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Preechakul¹,

Chatthee²,

Wizadwongsa³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Preechakul¹,

Chatthee²,

Wizadwongsa³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Each value is continuous and bounded. We use an action repeat of 2 to produce a better signal to the model (Hafner et al 2019). We keep the original reward, which incentivizes the agent to drive through as many tiles as possible.…”

Section: E1 Environmentsmentioning

confidence: 99%

“…Image-based environments: We implement PlaNet (Hafner et al 2019), which models the environment transition function using a latent dynamics model with deterministic and stochastic transition states; we refer interested readers to the original paper for details. PlaNet does not provide an uncertainty estimate because it only utilizes a single transition model.…”

Section: E2 Uncertainty Estimatorsmentioning

confidence: 99%

Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning

Ma¹,

Shen²,

Bastani³

et al. 2021

Preprint

View full text Add to dashboard Cite

Reinforcement Learning (RL) agents in the real world must satisfy safety constraints in addition to maximizing a reward objective. Model-based RL algorithms hold promise for reducing unsafe real-world actions: they may synthesize policies that obey all constraints using simulated samples from a learned model. However, imperfect models can result in realworld constraint violations even for actions that are predicted to satisfy all constraints. We propose Conservative and Adaptive Penalty (CAP), a model-based safe RL framework that accounts for potential modeling errors by capturing model uncertainty and adaptively exploiting it to balance the reward and the cost objectives. First, CAP inflates predicted costs using an uncertainty-based penalty. Theoretically, we show that policies that satisfy this conservative cost constraint are guaranteed to also be feasible in the true environment. We further show that this guarantees the safety of all intermediate solutions during RL training. Further, CAP adaptively tunes this penalty during training using true cost feedback from the environment. We evaluate this conservative and adaptive penalty-based approach for model-based safe RL extensively on state and image-based environments. Our results demonstrate substantial gains in sample-efficiency while incurring fewer violations than prior safe RL algorithms. Code is available at: https://github.com/Redrew/CAP

show abstract

“…On the other hand, several recent works incorporate the action (i.e., control command) of the agents in the latent representation [1, 13, 17-19, 29, 41, 42, 46]. While one can use the convolutional recurrent neural network to embed the entire past observations and actions to yield rich temporal information [1], most works first extract the low-dimensional latent dynamics model from the observation with actionconditioned SSM, and integrate the learned latent model into the agent's policy [29] or vision-based planning [19]. However, these approaches use a simple variational autoencoder to extract the latent state and thus cannot represent entity-wise interaction.…”

Section: Latent Dynamics Model From Visual Sequencementioning

confidence: 99%

GATSBI: Generative Agent-centric Spatio-temporal Object Interaction

Min

Bae

Lee

et al. 2021

Preprint

View full text Add to dashboard Cite

We present GATSBI, a generative model that can transform a sequence of raw observations into a structured latent representation that fully captures the spatio-temporal context of the agent's actions. In vision-based decision making scenarios, an agent faces complex high-dimensional observations where multiple entities interact with each other. The agent requires a good scene representation of the visual observation that discerns essential components and consistently propagates along the time horizon. Our method, GATSBI, utilizes unsupervised object-centric scene representation learning to separate an active agent, static background, and passive objects. GATSBI then models the interactions reflecting the causal relationships among decomposed entities and predicts physically plausible future states. Our model generalizes to a variety of environments where different types of robots and objects dynamically interact with each other. We show GATSBI achieves superior performance on scene decomposition and video prediction compared to its state-of-the-art counterparts.

show abstract

Learning Latent Dynamics for Planning from Pixels

Cited by 100 publications

References 0 publications

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation

Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning

GATSBI: Generative Agent-centric Spatio-temporal Object Interaction

Contact Info

Product

Resources

About