Evolving multimodal behavior with modular neural networks in Ms. Pac-Man

Abstract

Ms. Pac-Man is a challenging video game in which multiple modes of behavior are required to succeed: Ms. Pac-Man must escape ghosts when they are threats, and catch them when they are edible, in addition to eating all pills in each level. Past approaches to learning behavior in Ms. Pac-Man have treated the game as a single task to be learned using monolithic policy representations. In contrast, this paper uses a framework called Modular Multiobjective NEAT to evolve modular neural networks. Each module defines a separate policy; evolution discovers these policies and when to use them. The number of modules can be fixed or learned using a new version of a genetic operator, called Module Mutation, which duplicates an existing module that can then evolve to take on a distinct behavioral identity. Both the fixed modular networks and Module Mutation networks outperform traditional monolithic networks. More interestingly, the best modular networks dedicate modules to critical behaviors that do not follow the customary division of the game into chasing edible and escaping threatening ghosts.

Solving Interleaved and Blended Sequential Decision-Making Problems through Modular Neuroevolution

“…This game requires multimodal behavior because enemy agents are sometimes threats, and sometimes sources of points. Most of the modular architectures described above were applied to the original, blended version of Ms. PacMan [24], in which a mixture of edible and threat ghosts can be present at the same time. However, the interleaved version of the domain, in which such mixtures never occur, is new, and seeing these results side-by-side gives new insight into when and why certain types of modular networks are successful.…”

“…A similar approach is Module Mutation [23,24], which introduces groups of neurons rather than individual neurons. A single Module Mutation adds enough output neurons to define a new policy, plus an additional neuron to arbitrate between modules.…”

“…The variant of NEAT used in this paper is Modular Multiobjective NEAT 1 (MM-NEAT [24]), which distinguishes itself from NEAT by incorporating NSGA-II, and providing several methods for creating networks with multiple output modules, described next.…”

“…Different versions of Module Mutation exist [23,24]. This paper uses MM(D), for duplicate, because each new module duplicates the behavior of an existing module.…”

“…However, to succeed in such domains, agents need the freedom to discover their own task divisions. This paper builds on recent research in Ms. Pac-Man using evolved neural networks [24]. To support multiple tasks, networks have separate output modules to represent different behaviors.…”