Active Hierarchical Exploration with Stable Subgoal Representation Learning

Abstract

Goal-conditioned hierarchical reinforcement learning (HRL) serves as a successful approach to solving complex and temporally extended tasks. Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations. However, online subgoal representation learning exacerbates the non-stationary issue of HRL and introduces challenges for exploration in high-level policy learning. In this paper, we propose a state-specific regularization that stabilizes subgoal embeddings in well-explored areas while allowing representation updates in less explored state regions. Benefiting from this stable representation, we design measures of novelty and potential for subgoals, and develop an efficient hierarchical exploration strategy that actively seeks out new promising subgoals and states. Experimental results show that our method significantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards and further demonstrate the stability and efficiency of the subgoal representation learning of this work, which promotes superior policy learning.Preprint. Under review.

An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey

“…Figure 4b)), and whether they need a hand-made goal space (x,y) or an implicit curriculum of objectives. We can make two major observations: 1-methods that do not propose diverse goal-states require an implicit curriculum to learn the Ant-Maze task [Li et al 2021b;Nachum et al 2018] (Curriculum column); 2-contrastive representations seem crucial to avoid using a hand-defined goal space like the (x,y) coordinated (Goal space column) [Li et al 2021a;Nachum et al 2019a]. For methods in the "fixing the goal distribution", we did not find a representative and widely used evaluation protocol/environment among works.…”

“…HESS [Li et al 2021a] partitions the embedding space of LESSON and rewards with a variant of a count-based bonus (see Section 5). It improves exploration in a two-dimensional latent embedding but the size of partitions may not scale well if the agent considers more latent dimensions.…”

“…By configuring the time-extension of the representation, one may focus on different semantic parts of the state space. This can be seen in Section 3.2: 1-the agent can learn a very low level representation that provides skills that can manipulate torques of a creature [Aubret et al 2021]; 2-skills can also orientate an agent in a maze by extracting (x,y) coordinates from a complex state representation [Li et al 2021a]. While they do not try to combine and learn several representations at the same time, further works could consider separate different parts of states (e.g.…”

“…Considering a goal representation that changes over time introduces new issues for the agent. In this case, the goal-conditioned policy may be perturbed by the changes of inputs and may no longer be able to reach the goal [Li et al 2021a]. Current methods consider 1-developmental periods (unsupervised pre-training [Metzen and Kirchner 2013]); 2-to modify the representation every k-steps epochs [Pong et al 2020]; 3-to impose slowly changes of the representation [Li et al 2021a].…”

“…In this case, the goal-conditioned policy may be perturbed by the changes of inputs and may no longer be able to reach the goal [Li et al 2021a]. Current methods consider 1-developmental periods (unsupervised pre-training [Metzen and Kirchner 2013]); 2-to modify the representation every k-steps epochs [Pong et al 2020]; 3-to impose slowly changes of the representation [Li et al 2021a]. Further works may thoroughly investigate the relation and transitions between these methods since they can relate to the concept of critical periods [Hensch 2004;Konczak 2004].…”