Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Péré, Alexandre; Forestier, Sébastien; Sigaud, Olivier; Oudeyer, Pierre-Yves

doi:10.48550/arxiv.1803.00781

Cited by 26 publications

(36 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several authors have explored a pre-training stage, sometimes paired with fine-tuning, based on unsupervised representation learning. Péré et al (2018) and Laversanne-Finot et al ( 2018) employ a two-stage framework wherein unsupervised representation learning is used to learn a model of the observations from which to sample goals for control in simple simulated environments. Nair et al (2018) propose a similar approach in the context of model-free Q-learning applied to 3-dimensional simulations and robots.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Control Through Non-Parametric Discriminative Rewards

Warde-Farley¹,

Wiele²,

Kulkarni³

et al. 2018

Preprint

View full text Add to dashboard Cite

Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptuallyspecified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains -Atari, the DeepMind Control Suite and DeepMind Lab.

show abstract

Section: Related Workmentioning

confidence: 99%

Unsupervised Control Through Non-Parametric Discriminative Rewards

Warde-Farley¹,

Wiele²,

Kulkarni³

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Nevertheless, those methods have not considered the stability of the subgoal representation learning, which results in a non-stationary high-level learning environment. Other prior methods utilize a predefined or pretrained subgoal space [4,5,24,25] to keep the stability of the subgoal representation. However, those methods require taskspecific human knowledge or extra training data.…”

Section: Related Workmentioning

confidence: 99%

“…Goal-conditioned hierarchical reinforcement learning (HRL) has long demonstrated great potential to solve temporally extended tasks with sparse and delayed rewards [1][2][3][4][5], where higher-level policies periodically communicate subgoals to lower-level ones, and lower-level policies are intrinsically rewarded for reaching those subgoals. Early goal-conditioned HRL studies used a hand-designed subgoal space, such as positions of robots [6,7] or objects in images [8].…”

Section: Introductionmentioning

confidence: 99%

Active Hierarchical Exploration with Stable Subgoal Representation Learning

Liu¹,

Zhang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Goal-conditioned hierarchical reinforcement learning (HRL) serves as a successful approach to solving complex and temporally extended tasks. Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations. However, online subgoal representation learning exacerbates the non-stationary issue of HRL and introduces challenges for exploration in high-level policy learning. In this paper, we propose a state-specific regularization that stabilizes subgoal embeddings in well-explored areas while allowing representation updates in less explored state regions. Benefiting from this stable representation, we design measures of novelty and potential for subgoals, and develop an efficient hierarchical exploration strategy that actively seeks out new promising subgoals and states. Experimental results show that our method significantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards and further demonstrate the stability and efficiency of the subgoal representation learning of this work, which promotes superior policy learning.Preprint. Under review.

show abstract

“…We consider another way of comparing together the FD spaces, by using the KL-Coverage metric from [39], which was also used in the original AURORA paper [11], to check whether two FD spaces are similar to each other (possibly hinting at their representation capabilities). We extend the original definition to handle multicontainers scenarios:…”

Section: A Pairwise Kl-coveragementioning

confidence: 99%

Ensemble Feature Extraction for Multi-Container Quality-Diversity Algorithms

Cazenille

2021

Preprint

View full text Add to dashboard Cite

Quality-Diversity algorithms search for large collections of diverse and high-performing solutions, rather than just for a single solution like typical optimisation methods. They are specially adapted for multi-modal problems that can be solved in many different ways, such as complex reinforcement learning or robotics tasks. However, these approaches are highly dependent on the choice of feature descriptors (FDs) quantifying the similarity in behaviour of the solutions. While FDs usually needs to be hand-designed, recent studies have proposed ways to define them automatically by using feature extraction techniques, such as PCA or Auto-Encoders, to learn a representation of the problem from previously explored solutions.Here, we extend these approaches to more complex problems which cannot be efficiently explored by relying only on a single representation but require instead a set of diverse and complementary representations. We describe MC-AURORA, a Quality-Diversity approach that optimises simultaneously several collections of solutions, each with a different set of FDs, which are, in turn, defined automatically by an ensemble of modular auto-encoders. We show that this approach produces solutions that are more diverse than those produced by single-representation approaches.

show abstract

Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Cited by 26 publications

References 21 publications

Unsupervised Control Through Non-Parametric Discriminative Rewards

Unsupervised Control Through Non-Parametric Discriminative Rewards

Active Hierarchical Exploration with Stable Subgoal Representation Learning

Ensemble Feature Extraction for Multi-Container Quality-Diversity Algorithms

Contact Info

Product

Resources

About