Offline Reinforcement Learning from Images with Latent Space Models

Rafailov, Rafael; Yu, Tianhe; Rajeswaran, Aravind; Finn, Chelsea

doi:10.48550/arxiv.2012.11547

Cited by 3 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We showed the transfer capabilities of our algorithm to new tasks in a zero-shot imitation learning formulation. However, V-MAIL can in principle utilize any previously collected data for model-training, enabling potential applications in offline imitation learning in conjunction with offline RL algorithms like Rafailov et al [42].…”

Section: Discussionmentioning

confidence: 99%

“…Reinforcement learning from images is an inherently difficult task, since the agent needs to learn meaningful visual representations to support policy learning. A recent line of research [19,23,36,24,42] train a variational model of the image-based environment as an auxiliary task, either for representation learning only [19,36] or for additionally generating on-policy data by rolling out the model [24]. Our method builds upon these ideas, but unlike these prior works, considers the problem of learning from visual demonstrations without access to rewards.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Visual Adversarial Imitation Learning using Variational Models

Rafailov¹,

Yu²,

Rajeswaran³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Reward function specification, which requires considerable human effort and iteration, remains a major impediment for learning behaviors through deep reinforcement learning. In contrast, providing visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how to perform a task, and must learn to solve the task using the provided demonstrations and unsupervised environment interactions. This setting presents a number of challenges including representation learning for visual observations, sample complexity due to high dimensional spaces, and learning instability due to the lack of a fixed reward or learning signal. Towards addressing these challenges, we develop a variational model-based adversarial imitation learning (V-MAIL) algorithm. The model-based approach provides a strong signal for representation learning, enables sample efficiency, and improves the stability of adversarial training by enabling onpolicy learning. Through experiments involving several vision-based locomotion and manipulation tasks, we find that V-MAIL learns successful visuomotor policies in a sample-efficient manner, has better stability compared to prior work, and also achieves higher asymptotic performance. We further find that by transferring the learned models, V-MAIL can learn new tasks from visual demonstrations without any additional environment interactions. All results including videos can be found online at https://sites.google.com/view/variational-mail.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Visual Adversarial Imitation Learning using Variational Models

Rafailov¹,

Yu²,

Rajeswaran³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Model-free deep RL has been effective on short-horizon skills like object grasping [21,20,22], pushing and throwing [19,23,24], and multi-task learning [23,25] from images. Alternatively, model-based approaches [26,27], which explicitly learn the environment forward dynamics from images have also been employed for multi-task vision-based robotic manipulation tasks, either with planning algorithms [28,29,30,31,32,33,34,35] or for optimizing a parametric policy [36]. Unlike these prior works, our algorithm uses a model in conjunction with task-specific success classifiers and Q-functions to learn a wide range of skills suitable for sequencing into long-horizon tasks.…”

Section: Related Workmentioning

confidence: 99%

Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks

Wu¹,

Nair²,

Li³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we study the problem of learning a repertoire of lowlevel skills from raw images that can be sequenced to complete long-horizon visuomotor tasks. Reinforcement learning (RL) is a promising approach for acquiring short-horizon skills autonomously. However, the focus of RL algorithms has largely been on the success of those individual skills, more so than learning and grounding a large repertoire of skills that can be sequenced to complete extended multi-stage tasks. The latter demands robustness and persistence, as errors in skills can compound over time, and may require the robot to have a number of primitive skills in its repertoire, rather than just one. To this end, we introduce EMBR, a model-based RL method for learning primitive skills that are suitable for completing long-horizon visuomotor tasks. EMBR learns and plans using a learned model, critic, and success classifier, where the success classifier serves both as a reward function for RL and as a grounding mechanism to continuously detect if the robot should retry a skill when unsuccessful or under perturbations. Further, the learned model is task-agnostic and trained using data from all skills, enabling the robot to efficiently learn a number of distinct primitives. These visuomotor primitive skills and their associated pre-and post-conditions can then be directly combined with off-the-shelf symbolic planners to complete long-horizon tasks. On a Franka Emika robot arm, we find that EMBR enables the robot to complete three long-horizon visuomotor tasks at 85% success rate, such as organizing a desk, a cabinet, and drawers, which require sequencing up to 12 skills, involve 14 unique learned primitives, and demand generalization to novel objects.

show abstract

“…They can realize their bias by either estimating uncertainty quantification to the value function [7,9,10,16], using importance sampling based algorithms [21][22][23][24][25][26][27], explicitly constraining the learning policy to be close to the dataset [8,28], learning the conservative value function [11], using KL divergence [29,9,30] or MMD [10]. On the other hand, prior model-based offline RL methods [12][13][14][31][32][33][34][35][36][37] have studied in model-uncertainty quantification [12,13,38], representation learning [14], constraining the policy to imitate the behavioral policy [34], and using conservative estimation of value function [15]. Different from these works, we propose ROMI to investigate a new direction in the model-based offline RL, which will provide natural conservatism bias with maintaining superior model-based generalization benefits.…”

Section: Related Workmentioning

confidence: 99%

Offline Reinforcement Learning with Reverse Model-based Imagination

Wang¹,

Li²,

Jiang³

et al. 2021

Preprint

View full text Add to dashboard Cite

In offline reinforcement learning (offline RL), one of the main challenges is to deal with the distributional shift between the learning policy and the given dataset. To address this problem, recent offline RL methods attempt to introduce conservatism bias to encourage learning on high-confidence areas. Model-free approaches directly encode such bias into policy or value function learning using conservative regularizations or special network structures, but their constrained policy search limits the generalization beyond the offline dataset. Model-based approaches learn forward dynamics models with conservatism quantifications and then generate imaginary trajectories to extend the offline datasets. However, due to limited samples in offline dataset, conservatism quantifications often suffer from overgeneralization in out-of-support regions. The unreliable conservative measures will mislead forward model-based imaginations to undesired areas, leading to overaggressive behaviors. To encourage more conservatism, we propose a novel modelbased offline RL framework, called Reverse Offline Model-based Imagination (ROMI). We learn a reverse dynamics model in conjunction with a novel reverse policy, which can generate rollouts leading to the target goal states within the offline dataset. These reverse imaginations provide informed data augmentation for the model-free policy learning and enable conservative generalization beyond the offline dataset. ROMI can effectively combine with off-the-shelf model-free algorithms to enable model-based generalization with proper conservatism. Empirical results show that our method can generate more conservative behaviors and achieve state-of-the-art performance on offline RL benchmark tasks. * Equal contribution. † Work done while Guangxiang was a Ph.D. student at Tsinghua University.

show abstract

Offline Reinforcement Learning from Images with Latent Space Models

Cited by 3 publications

References 25 publications

Visual Adversarial Imitation Learning using Variational Models

Visual Adversarial Imitation Learning using Variational Models

Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks

Offline Reinforcement Learning with Reverse Model-based Imagination

Contact Info

Product

Resources

About