Training a spiking neuronal network model of visual-motor cortex to play a virtual racket-ball game using reinforcement learning

Abstract

Recent models of spiking neuronal networks have been trained to perform behaviors in static environments using a variety of learning rules, with varying degrees of biological realism. Most of these models have not been tested in dynamic visual environments where models must make predictions on future states and adjust their behavior accordingly. The models using these learning rules are often treated as black boxes, with little analysis on circuit architectures and learning mechanisms supporting optimal performance. Here we developed visual/motor spiking neuronal network models and trained them to play a virtual racket-ball game using several reinforcement learning algorithms inspired by the dopaminergic reward system. We systematically investigated how different architectures and circuit-motifs (feed-forward, recurrent, feedback) contributed to learning and performance. We also developed a new biologically-inspired learning rule that significantly enhanced performance, while reducing training time. Our models included visual areas encoding game inputs and relaying the information to motor areas, which used this information to learn to move the racket to hit the ball. Neurons in the early visual area relayed information encoding object location and motion direction across the network. Neuronal association areas encoded spatial relationships between objects in the visual scene. Motor populations received inputs from visual and association areas representing the dorsal pathway. Two populations of motor neurons generated commands to move the racket up or down. Model-generated actions updated the environment and triggered reward or punishment signals that adjusted synaptic weights so that the models could learn which actions led to reward. Here we demonstrate that our biologically-plausible learning rules were effective in training spiking neuronal network models to solve problems in dynamic environments. We used our models to dissect the circuit architectures and learning rules most effective for learning. Our model shows that learning mechanisms involving different neural circuits produce similar performance in sensory-motor tasks. In biological networks, all learning mechanisms may complement one another, accelerating the learning capabilities of animals. Furthermore, this also highlights the resilience and redundancy in biological systems.

Evolutionary and spike-timing-dependent reinforcement learning train spiking neuronal network motor control

“…Interestingly, the hyperparameter search revealed better preference when using the “ targeted RL both” paradigm. These findings suggest that targeted plasticity of specific motor areas could enhance the learning ability of the model, consistent with earlier findings((Anwar et al 2021; Patel et al 2019; Hazan et al 2018; Chadderdon et al 2012)).…”

“…This strategy considers that the underlying causality between pre and postsynaptic neurons and the associated reinforcement automatically changes only relevant synaptic connections. On top of the traditional STDP-RL approach, we used two recently developed versions of targeted reinforcement by selectively delivering reward and punishment to different subpopulations of the Motor population (EM) (Anwar et al 2021). In the first variation ( targeted RL main ), we delivered reward or punishment only to the neuronal subpopulation that generated the action (EM-LEFT or EM-RIGHT).…”

“…Traditionally, when SNN models are trained to perform a behavior using biologically inspired learning mechanisms, algorithms used are variations on either Spike Timing Dependent Plasticity(STDP) (Tavanaei et al 2019) or Evolutionary Strategies (Espinal et al 2014). For learning behaviors from the reinforcement learning domain, STDP can be extended to use reward modulated plasticity (Anwar et al 2021; Patel et al 2019; Hazan et al 2018; Chadderdon et al 2012; Neymotin et al 2013), an algorithm denoted Spike-timing dependent reinforcement learning (STDP-RL). In this work, we introduce a new Evolutionary Strategy variation, adapted from non-spiking neural networks (Salimans et al 2017).…”

Modernizing the NEURON Simulator for Sustainability, Portability, and Performance

“…NetPyNE (Dura-Bernal et al, 2019) is a high-level declarative NEURON wrapper used to develop a wide range of neural circuit models (Bryson et al, 2021; Pimentel et al, 2021; Volk et al, 2021; Ranieri et al, 2021; Metzner et al, 2020; Anwar et al, 2021; Sekiguchi et al, 2021; Dura-Bernal et al, 2022a,b; Borges et al, 2022; Romaro et al, 2021) 4 , and also as a resource for teaching neurobiology and computational neuroscience.…”

“…When running on a GPU, however, one must use the special executable to launch simulations due to limitations of the NVIDIA compiler toolchain when using OpenACC together with shared libraries. (Bryson et al, 2021;Pimentel et al, 2021;Volk et al, 2021;Ranieri et al, 2021;Metzner et al, 2020;Anwar et al, 2021;Sekiguchi et al, 2021;Dura-Bernal et al, 2022a,b;Borges et al, 2022;Romaro et al, 2021) 4 , and also as a resource for teaching neurobiology and computational neuroscience.…”