DisTop: Discovering a Topological representation to learn diverse and rewarding skills

Abstract

The optimal way for a deep reinforcement learning (DRL) agent to explore is to learn a set of skills that achieves a uniform distribution of states. Following this, we introduce DisTop, a new model that simultaneously learns diverse skills and focuses on improving rewarding skills. DisTop progressively builds a discrete topology of the environment using an unsupervised contrastive loss, a growing network and a goal-conditioned policy. Using this topology, a state-independent hierarchical policy can select where the agent has to keep discovering skills in the state space. In turn, the newly visited states allows an improved learnt representation and the learning loop continues. Our experiments emphasize that DisTop is agnostic to the ground state representation and that the agent can discover the topology of its environment whether the states are high-dimensional binary data, images, or proprioceptive inputs. We demonstrate that this paradigm is competitive on MuJoCo benchmarks with state-of-the-art algorithms on both single-task dense rewards and diverse skill discovery. By combining these two aspects, we show that DisTop achieves state-of-the-art performance in comparison with hierarchical reinforcement learning (HRL) when rewards are sparse. We believe DisTop opens new perspectives by showing that bottom-up skill discovery combined with representation learning can unlock the exploration challenge in DRL.Preprint. Under review.

An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey

“…Skew-fit manages to explore image-based environments very efficiently. As highlighted in [Aubret et al 2021], this ratio applied on a discrete number of skills, amount to rewards a Boltzmann goal-selection policy with:…”

“…It improves exploration in a two-dimensional latent embedding but the size of partitions may not scale well if the agent considers more latent dimensions. In contrast, DisTop [Aubret et al 2021] dynamically clusters a dynamic-aware embedding space using a variant of a Growing When Required [Marsland et al 2002]; they estimate the density of state according to how much its partition contains states and skew the distribution of sampled similarly to Skew-fit. HESS and DisTop demonstrate their ability to explore and navigate with an ant inside complex mazes without extrinsic rewards.…”

“…HESS and DisTop demonstrate their ability to explore and navigate with an ant inside complex mazes without extrinsic rewards. As shown in [Aubret et al 2021] (illustration in Figure 7c), it is also possible to use extrinsic rewards to weight the distribution of sampled state-goals.…”

“…Our surveyed works [Aubret et al 2021;Ermolov and Sebe 2020;Guo et al 2021;Li et al 2021b;Nachum et al 2019a] already propose to learn the representations using the slowness principle [Wiskott and Sejnowski 2002] which assumes temporally close states should be similarly represented. By configuring the time-extension of the representation, one may focus on different semantic parts of the state space.…”

“…By configuring the time-extension of the representation, one may focus on different semantic parts of the state space. This can be seen in Section 3.2: 1-the agent can learn a very low level representation that provides skills that can manipulate torques of a creature [Aubret et al 2021]; 2-skills can also orientate an agent in a maze by extracting (x,y) coordinates from a complex state representation [Li et al 2021a]. While they do not try to combine and learn several representations at the same time, further works could consider separate different parts of states (e.g.…”