Evolving neuromodulatory topologies for reinforcement learning-like problems

Abstract

Abstract-Environments with varying reward contingencies constitute a challenge to many living creatures. In such conditions, animals capable of adaptation and learning derive an advantage. Recent studies suggest that neuromodulatory dynamics are a key factor in regulating learning and adaptivity when reward conditions are subject to variability. In biological neural networks, specific circuits generate modulatory signals, particularly in situations that involve learning cues such as a reward or novel stimuli. Modulatory signals are then broadcast and applied onto target synapses to activate or regulate synaptic plasticity.Artificial neural models that include modulatory dynamics could prove their potential in uncertain environments when online learning is required. However, a topology that synthesises and delivers modulatory signals to target synapses must be devised. So far, only handcrafted architectures of such kind have been attempted. Here we show that modulatory topologies can be designed autonomously by artificial evolution and achieve superior learning capabilities than traditional fixed-weight or Hebbian networks. In our experiments, we show that simulated bees autonomously evolved a modulatory network to maximise the reward in a reinforcement learning-like environment.

Evolvability of Neuromodulated Learning for Robots

“…Recent results [5,8] suggest that the use of an implicit genetic encoding in place of a more conventional direct encoding endows the evolutionary process with several advantages, such as increased performance of the evolved structures, a more compact genome, and the possibility of complexification and simplification of the architecture via evolutionary duplication and mutation. From a robotics point of view the limitation of the results presented in previous experiments of neuromodulatory evolution such as [7,12,11] is the use of simplified tasks based on grid-like worlds and a choice between a finite, small set of actions. It is therefore important to extend those results and the insights gained through them, to more realistic robotic scenarios.…”

“…The recourse in [11] to artificial evolution for the synthesis of the neuromodulatory architectures is justified by the fact that it is difficult to hand-design both the underlying standard neural network and the additional neuromodulatory network that controls the learning of the former. In [12] we showed how an evolved neural architecture is typically more compact, while outperforming architectures designed by hand using the best neural networks and reinforcement learning practices. Recent results [5,8] suggest that the use of an implicit genetic encoding in place of a more conventional direct encoding endows the evolutionary process with several advantages, such as increased performance of the evolved structures, a more compact genome, and the possibility of complexification and simplification of the architecture via evolutionary duplication and mutation.…”

Real-time Hebbian Learning from Autoencoder Features for Control Tasks

“…Models augmented with neuromodulation have been shown to implement a variety of typical features of animal operant learning such as reinforcement of rewarding actions, extinction of unproductive actions, and behavior reversal (Soltoggio and Stanley, 2012;. The combination of Hebbian ANNs with neuromodulatory signals in recent years has especially inspired neuroevolution and artificial life researchers by opening up the possibility of evolving ANNs that can learn from a sequence of rewards over their lifetime (Soltoggio et al, 2008(Soltoggio et al, , 2007Soltoggio and Jones, 2009;Soltoggio and Stanley, 2012;Risi and Stanley, 2012;Coleman and Blair, 2012).…”

“…As a medium for adaptation and learning, neural plasticity has long captivated artificial life and related fields (Baxter, 1992;Floreano and Urzelai, 2000;Niv et al, 2002;Soltoggio et al, 2008Soltoggio et al, , 2007Soltoggio and Jones, 2009;Soltoggio and Stanley, 2012;Risi and Stanley, 2010;Risi et al, 2011;Risi and Stanley, 2012;Stanley et al, 2003;Coleman and Blair, 2012). Much of this body of research focuses on Hebbian-inspired rules that change the weights of connections in proportion to the correlation of source and target neuron activations (Hebb, 1949).…”

What is comparable in comparative cognition?

“…Relational categorization of concepts such as larger or smaller can be achieved using five neurons [83]. By using models of synaptic plasticity and neuromodulation [84], and again using in silico artificial evolution to find efficient circuits, it was possible to discover eight neuron reinforcement learning circuits that can learn the quality of reward associated with a resource and exhibit probability matching and risk-aversion phenomena predicted by optimal foraging theory [85]. Further work has examined predator-prey coevolution using similarly small neural networks to control behaviour, and even examined small neural circuits for altruism and cooperation [86].…”