When to Trust Your Model: Model-Based Policy Optimization

Jänner, M; Fu, Justin; Zhang, Marvin; Levine, Sergey

doi:10.48550/arxiv.1906.08253

Cited by 48 publications

(99 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model-based RL approaches typically alternate between fitting a predictive model of the environment dynamics/rewards and updating the control policies. The model can be used in various ways, such as execution-time planning [5,21], generating imaginary experiences for training the control policy [12,32]), etc. Our work is inspired by [7], which addresses the problem of error in long-horizon model dynamics prediction.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Multiagent Model-based Credit Assignment for Continuous Control

Han¹,

Lu²,

Michalak³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning (RL) has recently shown great promise in robotic continuous control tasks. Nevertheless, prior research in this vein center around the centralized learning setting that largely relies on the communication availability among all the components of a robot. However, agents in the real world often operate in a decentralised fashion without communication due to latency requirements, limited power budgets and safety concerns. By formulating robotic components as a system of decentralised agents, this work presents a decentralised multiagent reinforcement learning framework for continuous control. To this end, we first develop a cooperative multiagent PPO framework that allows for centralized optimisation during training and decentralised operation during execution. However, the system only receives a global reward signal which is not attributed towards each agent. To address this challenge, we further propose a generic game-theoretic credit assignment framework which computes agent-specific reward signals. Last but not least, we also incorporate a model-based RL module into our credit assignment framework, which leads to significant improvement in sample efficiency. We demonstrate the effectiveness of our framework on experimental results on Mujoco locomotion control tasks. For a demo video please visit: https://youtu.be/gFyVPm4svEY.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Atari and Go), problems in robotics often have high-dimensional continuous states and actions, and are often limited by real-world sample budgets [14]. To this end, prior research in robotic learning have developed RL algorithms capable of performing continuous control [10,11,17,28], and sample-efficient learning methods, e.g., [1,7,12].…”

Section: Introductionmentioning

confidence: 99%

Multiagent Model-based Credit Assignment for Continuous Control

Han¹,

Lu²,

Michalak³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…On the one hand stands the use of model predictive control in the engineering community where finely specified dynamics models are constructed by engineers and only a small number of parameters are fit with system identification to determine mass, inertia, joint stiffness, etc. On the other side of things stands the hands off approach taken in the RL community, where general and unstructured neural networks are used for both transition models [9,55,25] as well as policies and value functions [20]. The state and action spaces for these systems are highly complex with many diverse inputs like quaternions, joint angles, forces, torques that each transform in different ways under a symmetry transformation like a left-right reflection or a rotation.…”

Section: Approximate Symmetries In Reinforcement Learningmentioning

confidence: 99%

“…State of the art model based approaches on Mujoco tend to use an ensemble of small MLPs that predict the state transitions [9,55,25,2], without exploiting any structure of the state space. We evaluate test rollout predictions via the relative error of the state over different length horizons for the RPP model against an MLP, the method of choice.…”

Section: Better Transitionmentioning

confidence: 99%

Residual Pathway Priors for Soft Equivariance Constraints

Finzi¹,

Benton²,

Wilson³

2021

Preprint

View full text Add to dashboard Cite

There is often a trade-off between building deep learning systems that are expressive enough to capture the nuances of the reality, and having the right inductive biases for efficient learning. We introduce Residual Pathway Priors (RPPs) as a method for converting hard architectural constraints into soft priors, guiding models towards structured solutions, while retaining the ability to capture additional complexity. Using RPPs, we construct neural network priors with inductive biases for equivariances, but without limiting flexibility. We show that RPPs are resilient to approximate or misspecified symmetries, and are as effective as fully constrained models even when symmetries are exact. We showcase the broad applicability of RPPs with dynamical systems, tabular data, and reinforcement learning. In Mujoco locomotion tasks, where contact forces and directional rewards violate strict equivariance assumptions, the RPP outperforms baseline model-free RL agents, and also improves the learned transition models for model-based RL.

show abstract

“…Consequently, so far, many RL milestones have been achieved through simulating conspicuous amounts of experience and tuning for effective task-specific parameters (Mnih et al, 2013;Silver et al, 2017). Recent off-policy model-free (Chen et al, 2021) and model-based algorithms (Janner et al, 2019), pushed forward the state-of-the-art sample-efficiency on several benchmark simulation tasks (Brockman et al, 2016). We attribute such improvements to two main linked advances: more expressive models to capture uncertainty and better strategies to counteract detrimental biases from the learning process.…”

Section: Introductionmentioning

confidence: 99%

Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

Cetin¹,

Çeliktutan²

2021

Preprint

View full text Add to dashboard Cite

Popular off-policy deep reinforcement learning algorithms compensate for overestimation bias during temporal-difference learning by utilizing pessimistic estimates of the expected target returns. In this work, we propose a novel learnable penalty to enact such pessimism, based on a new way to quantify the critic's epistemic uncertainty. Furthermore, we propose to learn the penalty alongside the critic with dual TD-learning, a strategy to estimate and minimize the bias magnitude in the target returns. Our method enables us to accurately counteract overestimation bias throughout training without incurring the downsides of overly pessimistic targets. Empirically, by integrating our method and other orthogonal improvements with popular off-policy algorithms, we achieve state-of-the-art results in continuous control tasks from both proprioceptive and pixel observations.

show abstract

When to Trust Your Model: Model-Based Policy Optimization

Cited by 48 publications

References 10 publications

Multiagent Model-based Credit Assignment for Continuous Control

Multiagent Model-based Credit Assignment for Continuous Control

Residual Pathway Priors for Soft Equivariance Constraints

Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

Contact Info

Product

Resources

About