Online evolution of behavioural control on real robots is an open-ended approach to autonomous learning and adaptation: robots have the potential to automatically learn new tasks and to adapt to changes in environmental conditions, or to failures in sensors and/or actuators. However, studies have so far almost exclusively been carried out in simulation because evolution in real hardware has required several days or weeks to produce capable robots. In this article, we successfully evolve neural network-based controllers in real robotic hardware to solve two single-robot tasks and one collective robotics task. Controllers are evolved either from random solutions or from solutions pre-evolved in simulation. In all cases, capable solutions are found in a timely manner (1 h or less). Results show that more accurate simulations may lead to higher-performing controllers, and that completing the optimization process in real robots is meaningful, even if solutions found in simulation differ from solutions in reality. We furthermore demonstrate for the first time the adaptive capabilities of online evolution in real robotic hardware, including robots able to overcome faults injected in the motors of multiple units simultaneously, and to modify their behaviour in response to changes in the task requirements. We conclude by assessing the contribution of each algorithmic component on the performance of the underlying evolutionary algorithm.

A long-standing goal in artificial intelligence and robotics is synthesising agents that can effectively learn and adapt throughout their lifetime. One open-ended approach to behaviour learning in autonomous robots is online evolution , which is part of the evolutionary robotics field of research. In online evolution approaches, an evolutionary algorithm is executed on the robots during task execution, which enables continuous optimisation and adaptation of behaviour. Despite the potential for automatic behaviour learning, online evolution has not been widely adopted because it often requires several hours or days to synthesise solutions to a given task. In this respect, research in the field has failed to develop a prevalent algorithm able to effectively synthesise solutions to a large number of different tasks in a timely manner. Rather than focusing on a single algorithm, we argue for more general mechanisms that can combine the benefits of different algorithms to increase the performance of online evolution of robot controllers. We conduct a comprehensive assessment of a novel approach called online hyper-evolution (OHE). Robots executing OHE use the different sources of feedback information traditionally associated with controller evaluation to find effective evolutionary algorithms during task execution. First, we study two approaches: OHE-fitness, which uses the fitness score of controllers as the criterion to select promising algorithms over time, and OHE-diversity, which relies on the behavioural diversity of controllers for algorithm selection. We then propose a novel class of techniques called OHE-hybrid, which combine diversity and fitness to search for suitable algorithms. In addition to their effectiveness at selecting suitable algorithms, the different OHE approaches are evaluated for their ability to construct algorithms by controlling which algorithmic components should be employed for controller generation (e.g., mutation, crossover, among others), an unprecedented approach in evolutionary robotics. Results show that OHE (i) facilitates the evolution of controllers with high performance, (ii) can increase effectiveness at different stages of evolution by combining the benefits of multiple algorithms over time, and (iii) can be effectively applied to construct new algorithms during task execution. Overall, our study shows that OHE is a powerful new paradigm that allows robots to improve their learning process as they operate in the task environment.