Curiosity Search: Producing Generalists by Encouraging Individuals to Continually Explore and Acquire Skills throughout Their Lifetime

Abstract

Natural animals are renowned for their ability to acquire a diverse and general skill set over the course of their lifetime. However, research in artificial intelligence has yet to produce agents that acquire all or even most of the available skills in non-trivial environments. One candidate algorithm for encouraging the production of such individuals is Novelty Search, which pressures organisms to exhibit different behaviors from other individuals. However, we hypothesized that Novelty Search would produce sub-populations of specialists, in which each individual possesses a subset of skills, but no one organism acquires all or most of the skills. In this paper, we propose a new algorithm called Curiosity Search, which is designed to produce individuals that acquire as many skills as possible during their lifetime. We show that in a multiple-skill maze environment, Curiosity Search does produce individuals that explore their entire domain, while a traditional implementation of Novelty Search produces specialists. However, we reveal that when modified to encourage intra-life behavioral diversity, Novelty Search can produce organisms that explore almost as much of their environment as Curiosity Search, although Curiosity Search retains a significant performance edge. Finally, we show that Curiosity Search is a useful helper objective when combined with Novelty Search, producing individuals that acquire significantly more skills than either algorithm alone.

The evolution of intelligence in mammalian carnivores

“…We recommend that other workers adopt psychometric approaches to assess general intelligence in animals, so that we can start to understand general intelligence in a broader comparative context. Finally, we urge those interested in the evolution of animal brains and intelligence to consider these phenomena in the light of recent discoveries in scientific computation germane to the evolution of both modular [169] and general intelligence [170]. Computer simulations of evolving organisms have revealed that modularity evolves in neural and other networks as a by-product of selection for minimizing connection costs among nodes [169].…”

“…Stanton & Clune [170] have recently developed a new evolutionary algorithm yielding digital organisms that acquire as many skills as possible during their lifetime. We hope that the ability to evolve …”

The evolution of general intelligence

“…As a result, most current evolutionary algorithms produce domain-specific intelligence in machines that rarely possess more than a small set of skills, and they are thus suited to performing only tasks that demand that particular skill set. Although an intrinsic motivation to explore the environment has been imitated in artificial agents via machine learning (Schmidhuber 1991;Oudeyer et al 2007), the production of generalist learners within an evolutionary context remains highly problematic (Stanton & Clune 2016).…”

Procedural Content Generation through Quality Diversity

“…Indeed, QD draws inspiration from the idea of rewarding divergence to find the necessary steps towards high performing areas of the search space. In divergent search, artificial evolution is not guided by a fitness tied to the ultimate objective of the problem, but instead rewards directly the diversity of solutions, based on notions such as novelty [8], surprise [9] or curiosity [10]. QD combines the divergent properties of divergent search with localized convergence, as will be discussed below.…”