Towards Self-organized Control: Using Neural Cellular Automata to Robustly Control a Cart-pole Agent

Abstract

Neural cellular automata (Neural CA) are a recent framework used to model biological phenomena emerging from multicellular organisms. In these systems, artificial neural networks are used as update rules for cellular automata. Neural CA are end-to-end differentiable systems where the parameters of the neural network can be learned to achieve a particular task. In this work, we used neural CA to control a cart-pole agent. The observations of the environment are transmitted in input cells while the values of output cells are used as a readout of the system. We trained the model using deep-Q learning where the states of the output cells were used as the Q-value estimates to be optimized. We found that the computing abilities of the cellular automata were maintained over several hundreds of thousands of iterations, producing an emergent stable behavior in the environment it controls for thousands of steps. Moreover, the system demonstrated life-like phenomena such as a developmental phase, regeneration after damage, stability despite a noisy environment, and robustness to unseen disruption such as input deletion.

A Unified Substrate for Body-Brain Co-evolution

“…This update is only valid to the cells that are considered "alive", which are the ones that have their value in the body channel greater than 0.1 and their neighbors. This architecture is very similar to the ones in self-classifying MNIST [23] and in self-organized control of a cart-pole agent [31].…”

“…Our benchmark environments are complicated to deal using deep reinforcement learning due to the different number of inputs and outputs for the policy, as it was done for the cart-pole agent with NCA [31]. Therefore, we have chosen to use some derivative-free optimization methods.…”

“…A recent work by Bhatia et al [1] presents several virtual environments and also an algorithm for brain and body co-design with separated description methods for the morphology and control. In comparison with NCRS, our co-design algorithm consists of only one neural cellular automaton, which is known to have self-organizing and self-repair properties [19,11,31].…”

“…Morphology design is also present in other works [10,29,13]. The control phase is based on the NCA for controlling a cart-pole agent introduced by Variengien et al [31], but their NCA is trained using a reinforcement learning algorithm named deep-Q learning and the communication between NCA and environment happens in predefined cells. Our approach, NCRS, unifies these two methods by having two phases.…”

“…This type of encoding occurs in nature, and it facilitates the scaling of an organism because the information in the genome is much smaller than in the resulting phenotype. This is referred to as genomic bottleneck [35,31], and it implies that the genetic code of an organism compresses all the information to grow and maintain its body, and even a complex brain in some species.…”