Fast and stable MAP-Elites in noisy domains using deep grids

Abstract

Quality-Diversity optimisation algorithms enable the evolution of collections of both high-performing and diverse solutions. These collections offer the possibility to quickly adapt and switch from one solution to another in case it is not working as expected. It therefore finds many applications in real-world domain problems such as robotic control. However, QD algorithms, like most optimisation algorithms, are very sensitive to uncertainty on the fitness function, but also on the behavioural descriptors. Yet, such uncertainties are frequent in real-world applications. Few works have explored this issue in the specific case of QD algorithms, and inspired by the literature in Evolutionary Computation, mainly focus on using sampling to approximate the "true" value of the performances of a solution. However, sampling approaches require a high number of evaluations, which in many applications such as robotics, can quickly become impractical. In this work, we propose Deep-Grid MAP-Elites, a variant of the MAP-Elites algorithm that uses an archive of similar previously encountered solutions to approximate the performance of a solution. We compare our approach to previously explored ones on three noisy tasks: a standard optimisation task, the control of a redundant arm and a simulated Hexapod robot. The experimental results show that this simple approach is significantly more resilient to noise on the behavioural descriptors, while achieving competitive performances in terms of fitness optimisation, and being more sample-efficient than other existing approaches.

Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains

“…7. The results indicate that the value giving the best performance on average across environments is N = 32. of the grid D. The original Deep-grid paper [15] was using D = 50, we thus compare value similar values. However, as we are working with hardware-accelerated libraries we only consider powers of 2: 32, 64, 128.…”

“…3) Deep-Grid: Deep-Grid MAP-Elites [15] is an alternative Implicit sampling approach to Uncertain QD. It works by adding a depth to the MAP-Elites grid to keep D solutions per cell, so that each cell sub-population empirically estimates the distribution of the elite's fitness and descriptor.…”

“…We perform our comparison on the tasks suite illustrated in Table II, taken from [10], [15], [31] and chosen for their diversity of controller architectures, noise structures, and complexity. Among these tasks, the Walker and Ant tasks are controlled in closed-loop and the Arm and the Hexapod in open-loop.…”

Gradient-Informed Quality Diversity for the Illumination of Discrete Spaces

“…Application specific methods were developed to apply qd algorihtms to noisy domains [13] or when multiple objectives are at stake [47]. Reinforcement Learning is one of the most popular applications of qd [3] and some methods try to incorporate diversity directly in RL algorithms [42,46].…”