Search on the Replay Buffer: Bridging Planning and Reinforcement Learning

Eysenbach, Benjamin; Salakhutdinov, Ruslan; Levine, Sergey

doi:10.48550/arxiv.1906.05253

Cited by 8 publications

(20 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This can be a limitation in practice, especially in robotic domains, as any interaction with the environment requires robot time, and exploring a new environment can be challenging (indeed, Savinov et al 2018 applied manual exploration). In addition, similarly to Eysenbach et al (2019), we found that training the connectivity classifier as proposed by Savinov et al (2018) requires extensive hyperparameter tuning.…”

Section: Hallucinative Topological Memorymentioning

confidence: 84%

“…Additionally, similar to (Eysenbach et al, 2019), we find that training the graph connectivity classifier as originally proposed by (Savinov et al, 2018) requires extensive manual tuning. We replace the vanilla classifier used in SPTM with an energy-based model that employs a contrastive loss.…”

Section: Introductionmentioning

confidence: 85%

“…In practice, this leads to three important problems. First, this method is known to be sensitive to the choice of positive and negative labeling (Eysenbach et al, 2019). Second, training data are required to be long, noncyclic trajectories for a high likelihood of sampling 'true' negative samples.…”

Section: Learning the Connectivity Via Contrastive Lossmentioning

confidence: 99%

See 2 more Smart Citations

Hallucinative Topological Memory for Zero-Shot Visual Planning

Liu,

Kurutach,

Tung

et al. 2020

Preprint

View full text Add to dashboard Cite

In visual planning (VP), an agent learns to plan goal-directed behavior from observations of a dynamical system obtained offline, e.g., images obtained from self-supervised robot interaction. Most previous works on VP approached the problem by planning in a learned latent space, resulting in low-quality visual plans, and difficult training algorithms. Here, instead, we propose a simple VP method that plans directly in image space and displays competitive performance. We build on the semi-parametric topological memory (SPTM) method: image samples are treated as nodes in a graph, the graph connectivity is learned from image sequence data, and planning can be performed using conventional graph search methods. We propose two modifications on SPTM. First, we train an energy-based graph connectivity function using contrastive predictive coding that admits stable training. Second, to allow zero-shot planning in new domains, we learn a conditional VAE model that generates images given a context of the domain, and use these hallucinated samples for building the connectivity graph and planning. We show that this simple approach significantly outperform the SOTA VP methods, in terms of both plan interpretability and success rate when using the plan to guide a trajectory-following controller. Interestingly, our method can pick up nontrivial visual properties of objects, such as their geometry, and account for it in the plans.

show abstract

Section: Hallucinative Topological Memorymentioning

confidence: 84%

Section: Introductionmentioning

confidence: 85%

See 1 more Smart Citation

Hallucinative Topological Memory for Zero-Shot Visual Planning

Liu,

Kurutach,

Tung

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Pathak et al (2018) use an inverse model with forward consistency to learn from novelty seeking behavior, but lacks convergence guarantees and requires learning a complex inverse model. Semi-parametric methods (Savinov et al, 2018;Eysenbach et al, 2019) learn a policy similar to ours but do so by building a connectivity graph over the visited states in order to navigate environments, which requires large memory storage and computation time that increases with the number of states.…”

Section: Related Workmentioning

confidence: 99%

Learning to Reach Goals via Iterated Supervised Learning

Ghosh¹,

Gupta²,

Reddy³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Imitation learning algorithms provide a simple and straightforward approach for training control policies via supervised learning. By maximizing the likelihood of good actions provided by an expert demonstrator, supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of reinforcement learning, at the cost of requiring an expert demonstrator to provide the demonstrations. In this paper, we ask: can we take insights from imitation learning to design algorithms that can effectively acquire optimal policies from scratch without any expert demonstrations? The key observation that makes this possible is that, in the multi-task setting, trajectories that are generated by a suboptimal policy can still serve as optimal examples for other tasks. In particular, when tasks correspond to different goals, every trajectory is a successful demonstration for the goal state that it actually reaches. We propose a simple algorithm for learning goal-reaching behaviors without any demonstrations, complicated user-provided reward functions, or complex reinforcement learning methods. Our method simply maximizes the likelihood of actions the agent actually took in its own previous rollouts, conditioned on the goal being the state that it actually reached. Although related variants of this approach have been proposed previously in imitation learning with demonstrations, we show how this approach can effectively learn goal-reaching policies from scratch. We present a theoretical result linking self-supervised imitation learning and reinforcement learning, and empirical results showing that it performs competitively with more complex reinforcement learning methods on a range of challenging goal reaching problems, while yielding advantages in terms of stability and use of offline data.

show abstract

“…We experimentally compare to RL-based distance learning methods, and show that DDL attains substantially better results, especially with complex observations. Another line of prior work uses a learned distance to build a search graph over a set of visited states (Savinov et al, 2018;Eysenbach et al, 2019), which can then be used to plan to reach new states via the shortest path.…”

Section: Related Workmentioning

confidence: 99%

Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery

Hartikainen¹,

Geng²,

Haarnoja³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning requires manual specification of a reward function to learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even infeasible unless the reward function is shaped so as to provide a smooth gradient towards a successful outcome. This shaping is difficult to specify by hand, particularly when the task is learned from raw observations, such as images. In this paper, we study how we can automatically learn dynamical distances: a measure of the expected number of time steps to reach a given goal state from any other state. These dynamical distances can be used to provide well-shaped reward functions for reaching new goals, making it possible to learn complex tasks efficiently. We show that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the environment is used to learn the dynamical distances, while a small amount of preference supervision is used to determine the task goal, without any manually engineered reward function or goal examples. We evaluate our method both on a real-world robot and in simulation. We show that our method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. Videos of the learned skills can be found on the project website: https://sites.google.com/view/dynamical-distance-learning.

show abstract

Search on the Replay Buffer: Bridging Planning and Reinforcement Learning

Cited by 8 publications

References 39 publications

Hallucinative Topological Memory for Zero-Shot Visual Planning

Hallucinative Topological Memory for Zero-Shot Visual Planning

Learning to Reach Goals via Iterated Supervised Learning

Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery

Contact Info

Product

Resources

About