Unsupervised Learning and Exploration of Reachable Outcome Space

Paolo, Giuseppe; Laflaquière, Alban; Coninx, Alexandre; Doncieux, Stéphane

doi:10.1109/icra40945.2020.9196819

Cited by 29 publications

(31 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The computing resources of the robotic platform also limited the training of the representation in the online experiment: the representation was trained using the first thousand examples but was not updated further on. In future work it would be interesting to continuously train the representation during exploration as experimented with in other works in simulated environments (Cully, 2019 ; Paolo et al, 2020 ; Reinke et al, 2020 ).…”

Section: Discussionmentioning

confidence: 99%

“…Other works have also leveraged the power of deep representation algorithms in the context of population-based divergent search algorithms (Cully, 2019 ; Paolo et al, 2020 ). In these works, a low dimensional representation of the environment is learned during exploration and serves as behavioral descriptors for Novelty-Search or Quality-Diversity exploration algorithms.…”

Section: Related Workmentioning

confidence: 99%

“…Thus, exploration algorithms are often evaluated using the coverage of the state space. Such measures were also used in a similar context for population-based divergent search algorithms (Cully, 2019 ; Paolo et al, 2020 ).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Intrinsically Motivated Exploration of Learned Goal Spaces

2021

View full text Add to dashboard Cite

Finding algorithms that allow agents to discover a wide variety of skills efficiently and autonomously, remains a challenge of Artificial Intelligence. Intrinsically Motivated Goal Exploration Processes (IMGEPs) have been shown to enable real world robots to learn repertoires of policies producing a wide range of diverse effects. They work by enabling agents to autonomously sample goals that they then try to achieve. In practice, this strategy leads to an efficient exploration of complex environments with high-dimensional continuous actions. Until recently, it was necessary to provide the agents with an engineered goal space containing relevant features of the environment. In this article we show that the goal space can be learned using deep representation learning algorithms, effectively reducing the burden of designing goal spaces. Our results pave the way to autonomous learning agents that are able to autonomously build a representation of the world and use this representation to explore the world efficiently. We present experiments in two environments using population-based IMGEPs. The first experiments are performed on a simple, yet challenging, simulated environment. Then, another set of experiments tests the applicability of those principles on a real-world robotic setup, where a 6-joint robotic arm learns to manipulate a ball inside an arena, by choosing goals in a space learned from its past experience.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Intrinsically Motivated Exploration of Learned Goal Spaces

2021

View full text Add to dashboard Cite

show abstract

“…For this assumption to hold, the space in which the variations are observed needs to be carefully chosen. This is the motivation for recent approaches which propose to automatically build it [4,28]. Regardless of its origin, this space needs to make sense with respect to the kind of task the robot could have to achieve.…”

Section: Evolvability: Definition and Estimationmentioning

confidence: 99%

Novelty search makes evolvability inevitable

Doncieux¹,

Paolo²,

Laflaquière

et al. 2020

Proceedings of the 2020 Genetic and Evolutionary Computation Conference

Self Cite

View full text Add to dashboard Cite

Evolvability is an important feature that impacts the ability of evolutionary processes to find interesting novel solutions and to deal with changing conditions of the problem to solve. The estimation of evolvability is not straight-forward and is generally too expensive to be directly used as selective pressure in the evolutionary process. Indirectly promoting evolvability as a side effect of other easier and faster to compute selection pressures would thus be advantageous. In an unbounded behavior space, it has already been shown that evolvable individuals naturally appear and tend to be selected as they are more likely to invade empty behavior niches. Evolvability is thus a natural byproduct of the search in this context. However, practical agents and environments often impose limits on the reachable behavior space. How do these boundaries impact evolvability? In this context, can evolvability still be promoted without explicitly rewarding it? We show that Novelty Search implicitly creates a pressure for high evolvability even in bounded behavior spaces, and explore the reasons for such a behavior. More precisely we show that, throughout the search, the dynamic evaluation of novelty rewards individuals which are very mobile in the behavior space, which in turn promotes evolvability. CCS CONCEPTS• Computing methodologies → Search methodologies; Evolutionary robotics; Neural networks;

show abstract

“…Novelty Search (NS) [22], Surprise Search [20] and Curiosity Search [12] are among divergent methods that define a behavior space as a proxy for conducting the search. Such a space can be either hand-engineered or learned [11,13,24,26]. In this paper, we focus on NS, but the proposed method can also be applied to other search approaches.…”

Section: Introductionmentioning

confidence: 99%

Br-Ns

Salehi

Coninx

Doncieux

2021

Proceedings of the Genetic and Evolutionary Computation Conference

Self Cite

View full text Add to dashboard Cite

As open-ended learning based on divergent search algorithms such as Novelty Search (NS) draws more and more attention from the research community, it is natural to expect that its application to increasingly complex real-world problems will require the exploration to operate in higher dimensional Behavior Spaces (BSs) which will not necessarily be Euclidean. Novelty Search traditionally relies on k-nearest neighbours search and an archive of previously visited behavior descriptors which are assumed to live in a Euclidean space. This is problematic because of a number of issues. On one hand, Euclidean distance and Nearest-neighbour search are known to behave differently and become less meaningful in high dimensional spaces. On the other hand, the archive has to be bounded since, memory considerations aside, the computational complexity of finding nearest neighbours in that archive grows linearithmically with its size. A sub-optimal bound can result in "cycling" in the behavior space, which inhibits the progress of the exploration. Furthermore, the performance of NS depends on a number of algorithmic choices and hyperparameters, such as the strategies to add or remove elements to the archive and the number of neighbours to use in k-nn search. In this paper, we discuss an alternative approach to novelty estimation, dubbed Behavior Recognition based Novelty Search (BR-NS), which does not require an archive, makes no assumption on the metrics that can be defined in the behavior space and does not rely on nearest neighbours search. We conduct experiments to gain insight into its feasibility and dynamics as well as potential advantages over archive-based NS in terms of time complexity. CCS CONCEPTS• Computing methodologies → Search methodologies; Evolutionary robotics; Neural networks;

show abstract

Unsupervised Learning and Exploration of Reachable Outcome Space

Cited by 29 publications

References 21 publications

Intrinsically Motivated Exploration of Learned Goal Spaces

Intrinsically Motivated Exploration of Learned Goal Spaces

Novelty search makes evolvability inevitable

Br-Ns

Contact Info

Product

Resources

About