Evolutionary and spike-timing-dependent reinforcement learning train spiking neuronal network motor control

Abstract

Biological learning operates at multiple interlocking timescales, from long evolutionary stretches down to the relatively short time span of an individual’s life. While each process has been simulated individually as a basic learning algorithm in the context of spiking neuronal networks (SNNs), the integration of the two has remained limited. In this study, we first train SNNs separately using individual model learning using spike-timing dependent reinforcement learning (STDP-RL) and evolutionary (EVOL) learning algorithms to solve the CartPole reinforcement learning (RL) control problem. We then develop an interleaved algorithm inspired by biological evolution that combines the EVOL and STDP-RL learning in sequence.We use the NEURON simulator with NetPyNE to create an SNN interfaced with the CartPole environment from OpenAI’s Gym. In CartPole, the goal is to balance a vertical pole by moving left/right on a 1-D plane. Our SNN contains multiple populations of neurons organized in three layers: sensory layer, association/hidden layer, and motor layer, where neurons are connected by excitatory (AMPA/NMDA) and inhibitory (GABA) synapses. Association and motor layers contain one excitatory (E) population and two inhibitory (I) populations with different synaptic time constants. Each neuron is an event-based integrate-and-fire model with plastic connections between excitatory neurons. In our SNN, the environment activates sensory neurons tuned to specific features of the game state. We split the motor population into subsets representing each movement choice. The subset with more spiking over an interval determines the action.During STDP-RL, we supply intermediary evaluations (reward/punishment) of each action by judging the effectiveness of a move (e.g., moving the CartPole to a balanced position). During EVOL, updates consist of adding together many random perturbations of the connection weights. Each set of random perturbations is weighted by the total episodic reward it achieves when applied independently. We evaluate the performance of each algorithm after training and through the creation of sensory/motor action maps that delineate the network’s transformation of sensory inputs into higher-order representations and eventual motor decisions. Both EVOL and STDP-RL training produce SNNs capable of moving the cart left and right and keeping the pole vertical. Compared to the STDP-RL and EVOL algorithms operating on their own, our interleaved training paradigm produced enhanced robustness in performance, with different strategies revealed through analysis of the sensory/motor mappings. Analysis of synaptic weight matrices also shows distributed vs clustered representations after the EVOL and STDP-RL algorithms, respectively. These weight differences also manifest as diffuse vs synchronized firing patterns. Our modeling opens up new capabilities for SNNs in RL and could serve as a testbed for neurobiologists aiming to understand multi-timescale learning mechanisms and dynamics in neuronal circuits.

Training a spiking neuronal network model of visual-motor cortex to play a virtual racket-ball game using reinforcement learning

“…Inspired by these mechanisms, Najarro and Risi further showed added advantage to the hebbian and other biological plasticity mechanisms. Including additional plasticity mechanisms and randomizing synaptic connections and weights might lead to even better performance as such capabilities of multi-timescale learning in SNNs for learning visual-motor behaviors have been shown in another study [ 95 ].…”

“…could use different types of dopamine receptors), but we are not aware of any direct experimental evidence of how associations between different rewards and respective motor actions take place at multiple time scales. Combining different timescales of reward/behavior in parallel, or independently [ 95 ] has also been shown to enhance performance. The time-scales underlying other learning mechanisms (such as homeostatic plasticity, sleep consolidation, etc) could provide additional benefits [ 105 ].…”

Training a spiking neuronal network model of visual-motor cortex to play a virtual racket-ball game using reinforcement learning

“…could use different types of dopamine receptors), but we are not aware of any direct experimental evidence of how associations between different rewards and respective motor actions take place at multiple time scales. Combining different timescales of reward/behavior in parallel, or independently [96] has also been shown to enhance performance. The time-scales underlying other learning mechanisms (such as homeostatic plasticity, sleep consolidation, etc) could provide additional benefits [106].…”

“…Inspired by these mechanisms, Najarro and Risi further showed added advantage to the hebbian and other biological plasticity mechanisms. Including additional plasticity mechanisms and randomizing synaptic connections and weights might lead to even better performance as such capabilities of multi-timescale learning in SNNs for learning visual-motor behaviors have been shown in another study [96].…”

Data-driven multiscale model of macaque auditory thalamocortical circuits reproduces in vivo dynamics

“…Automated optimization methods have been previously used for simpler networks (e.g. recurrent point-neuron spiking networks) (Nicola and Clopath 2017; Sussillo and Abbott 2009; Dura-Bernal et al 2017; Carlson et al 2014; Hasegan et al 2021). However, optimization of large-scale biophysically-detailed networks typically requires expert-guided parameter adjustments (Bezaire et al 2016; Markram et al 2015), for example through parameter sweeps (grid search) (Billeh et al 2020).…”