Sample and time efficient policy learning with CMA-ES and Bayesian Optimisation

Abstract

In evolutionary robot systems where morphologies and controllers of real robots are simultaneously evolved, it is clear that there is likely to be requirements to refine the inherited controller of a 'newborn' robot in order to better align it to its newly generated morphology. This can be accomplished via a learning mechanism applied to each individual robot: for practical reasons, such a mechanism should be both sample and time-efficient. In this paper, We investigate two ways to improve the sample and time efficiency of the well-known learner CMA-ES on navigation tasks. The first approach combines CMA-ES with Novelty Search, and includes an adaptive restart mechanism with increasing population size. The second bootstraps CMA-ES using Bayesian Optimisation, known for its sample efficiency. Results using two robots built with the ARE project's modules and four environments show that novelty reduces the number of samples needed to converge, as does the custom restart mechanism; the latter also has better sample and time efficiency than the hybridised Bayesian/Evolutionary method.

Bootstrapping Artificial Evolution to Design Robots for Autonomous Fabrication

“…First, a novelty search algorithm is used to evolve a large list of robots, from which a subset of robots is selected to form a bootstrap population. Secondly, a learning algorithm (NIP-ES, [26]) is applied to the bootstrap population to learn a controller for these robots, to gain insight into the potential capabilities of the bootstrap population. Finally, the bootstrap population (of body plans) is used to form the initial population of an evolutionary algorithm by pairing each body plan with a random CPPN that generates a controller.…”

“…In these experiments, the Novelty-driven Increasing Population Evolutionary Strategy (NIP-ES) algorithm is used to train a controller for each robot in the bootstrap population, based on our previous work [26]. The NIP-ES algorithm is a modified version of the increasing-population Covariance Matrix Adaptation Evolutionary Strategy (IPOP-CMAES) [27].…”

“…NIP-ES has custom stopping criteria and also includes a novelty score as an additional objective. Complete details of this method, including the stopping criteria and parameter values, can be found in [26] Learning finishes for a single robot when: (a) the budget of 1000 evaluations has been reached, or (b) the robot has reached the target, or (c) the robot has not moved in 50 evaluations. The robot has reached the target when the robot is 0.141 m away from the target which is equal to a fitness of 0.95 (the fitness is normalised using the maximum distance possible in the arena, namely the diagonal of the arena: 2.82 m).…”

Hardware Design for Autonomous Robot Evolution

“…10. These body plans are being used to test and develop some of the hardware, such as the Robot Fabricator [4], and software, such as selecting and developing the controller architecture and learning algorithms which will eventually be applied to evolved robots [16]. They are therefore designed to require different controllers, with a maze solving task in mind.…”

Evolution of Diverse, Manufacturable Robot Body Plans

“…Given that the search-space of body plans (morphologies) and controllers is very large, and printing/assembly is both time-consuming and expensive, we intend to bootstrap our evolutionary process by starting with a diverse population of manufacturable robots. For instance, it was shown by Le Goff et al [6] that for only learning it takes at least a couple hundred of evaluations to produce controllers for ARE-robots to solve tasks.…”