C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks

Abstract

Goal-conditioned reinforcement learning (RL) can solve tasks in a wide range of domains, including navigation and manipulation, but learning to reach distant goals remains a central challenge to the field. Learning to reach such goals is particularly hard without any offline data, expert demonstrations, and reward shaping. In this paper, we propose an algorithm to solve the distant goal-reaching task by using search at training time to automatically generate a curriculum of intermediate states. Our algorithm, Classifier-Planning (C-Planning), frames the learning of the goal-conditioned policies as expectation maximization: the E-step corresponds to planning an optimal sequence of waypoints using graph search, while the M-step aims to learn a goal-conditioned policy to reach those waypoints. Unlike prior methods that combine goal-conditioned RL with graph search, ours performs search only during training and not testing, significantly decreasing the compute costs of deploying the learned policy. Empirically, we demonstrate that our method is more sample efficient that prior methods. Moreover, it is able to solve very long horizons manipulation and navigation tasks, tasks that prior goalconditioned methods and methods based on graph search fail to solve. 1 * Equal contribution. 1 Code and videos of our results: https://ben-eysenbach.github.io/c-planning/

Divide & Conquer Imitation Learning

“…For instance, in Hierarchical RL, a highlevel policy may construct a sequence of goals which must be successively achieved by a low-level goal-conditioned policy [4,5,6,7,8,9]. Other approaches combine goal-conditioned value learning and goal-level planning to achieve distant goals [10,11,12,13,14]. However, in order to exploit an accurate value function, it should be noted that these planning methods are based on at least one of the two following assumptions: a dense and informative reward function is available, or an assumption of resetting in hard-to-attain states is required.…”