RM-SORN: a reward-modulated self-organizing recurrent neural network

Aswolinskiy, Witali; Pipa, Gordon

doi:10.3389/fncom.2015.00036

Cited by 13 publications

(12 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With these three rules at play, we exposed the model to patterned sensory stimulation that imitated retinotopic inputs from the eye ( Figure 7A). We hypothesized that during collision avoidance in real tadpoles, STDP-driven changes in the tectum may be controlled and amplified by a global learning signal that arrives if collision avoidance was unsuccessful (Savin and Triesch, 2014;Aswolinskiy and Pipa, 2015), and originates either in dimming receptors in the retina (Baranauskas et al, 2012), or in mechanosensory systems of the hindbrain Felch et al, 2016;Truszkowski et al, 2017). This approach is known as the eligibility trace model, in which changes in synaptic weights do not happen immediately, but are first "remembered" by each cell as potential changes, and are only implemented in response to a timely reinforcement signal (Seung, 2003).…”

Section: Developmental Modelmentioning

confidence: 99%

Graph analysis of looming-selective networks in the tectum, and its replication in a simple computational model

Khakhalin

2019

Preprint

View full text Add to dashboard Cite

Looming stimuli evoke behavioral responses in most sighted animals, yet the mechanisms of looming detection in vertebrates are poorly understood. Here we hypothesize that looming detection in the tectum may rely on spontaneous emergence of synfire chains: groups of neurons connected to each other in the same sequence in which they are activated during a loom. We then test some specific consequences of this hypothesis. First, we use high-speed calcium imaging to reconstruct connectivity of small networks within the tectum of Xenopus tadpoles. We report that reconstructed directed graphs are clustered and hierarchical, that their modularity increases in development, and that looming-selective cells tend to act as activation sinks within these graphs. Second, we describe spontaneous emergence of looming selectivity in a computational developmental model of the tectum, governed by both synaptic and intrinsic plasticity, and driven by structured visual inputs. We show that synfire chains contribute to looming detection in the model; that structured inputs are critical for the emergence of selectivity, and that biological tectal networks follow most, but not all predictions of the model. Finally, we propose a conceptual scheme for understanding the emergence and fine-tuning of collision detection in developing aquatic animals. 14(4):505.

show abstract

Section: Developmental Modelmentioning

confidence: 99%

Graph analysis of looming-selective networks in the tectum, and its replication in a simple computational model

Khakhalin

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…This finding has led to numerous studies investigating the role of synaptic scaling in controlling neural network activity (Turrigiano et al, 1998 ; Turrigiano and Nelson, 2000 ; Turrigiano, 2008 ) and in stabilizing other plasticity mechanisms (van Rossum et al, 2000 ; Stellwagen and Malenka, 2006 ; Tetzlaff, 2011 ; Toyoizumi et al, 2014 ). Indeed, synaptic scaling has proven successful in stabilizing activity in recurrent neural networks (Lazar et al, 2009 ; Remme and Wadman, 2012 ; Zenke et al, 2013 ; Effenberger and Jost, 2015 ; Miner and Triesch, 2016 ). However, these studies either used synaptic scaling as the sole homeostatic mechanism (Remme and Wadman, 2012 ; Zenke et al, 2013 ) or resorted to a variant of synaptic scaling where the scaling is not dynamically determined through a control loop using a particular target activity, but rather by a fixed multiplicative normalization rule (Lazar et al, 2009 ; Effenberger and Jost, 2015 ; Miner and Triesch, 2016 ).…”

Section: Introductionmentioning

confidence: 99%

Local Homeostatic Regulation of the Spectral Radius of Echo-State Networks

Schubert

Gros²

2021

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Recurrent cortical networks provide reservoirs of states that are thought to play a crucial role for sequential information processing in the brain. However, classical reservoir computing requires manual adjustments of global network parameters, particularly of the spectral radius of the recurrent synaptic weight matrix. It is hence not clear if the spectral radius is accessible to biological neural networks. Using random matrix theory, we show that the spectral radius is related to local properties of the neuronal dynamics whenever the overall dynamical state is only weakly correlated. This result allows us to introduce two local homeostatic synaptic scaling mechanisms, termed flow control and variance control, that implicitly drive the spectral radius toward the desired value. For both mechanisms the spectral radius is autonomously adapted while the network receives and processes inputs under working conditions. We demonstrate the effectiveness of the two adaptation mechanisms under different external input protocols. Moreover, we evaluated the network performance after adaptation by training the network to perform a time-delayed XOR operation on binary sequences. As our main result, we found that flow control reliably regulates the spectral radius for different types of input statistics. Precise tuning is however negatively affected when interneural correlations are substantial. Furthermore, we found a consistent task performance over a wide range of input strengths/variances. Variance control did however not yield the desired spectral radii with the same precision, being less consistent across different input strengths. Given the effectiveness and remarkably simple mathematical form of flow control, we conclude that self-consistent local control of the spectral radius via an implicit adaptation scheme is an interesting and biological plausible alternative to conventional methods using set point homeostatic feedback controls of neural firing.

show abstract

“…RL has a long tradition in the field of machine learning which has led to several powerful algorithms, such as SARSA and Q-learning (Watkins, 1989). Similarly, a large variety of neurobiological models have been proposed in recent years (Izhikevich, 2007; Potjans et al, 2009, 2011; Urbanczik and Senn, 2009; Vasilaki et al, 2009; Frémaux et al, 2010; Frémaux et al, 2013; Jitsev et al, 2012; Friedrich et al, 2014; Rasmussen and Eliasmith, 2014; Aswolinskiy and Pipa, 2015; Baladron and Hamker, 2015; Rombouts et al, 2015; Friedrich and Lengyel, 2016; Rueckert et al, 2016). However, only a small proportion of these rely on publicly available simulators and all of them employ custom built environments.…”

Section: Introductionmentioning

confidence: 99%

A Closed-Loop Toolchain for Neural Network Simulations of Learning Autonomous Agents

Jordan

Weidel

Morrison

2019

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Neural network simulation is an important tool for generating and evaluating hypotheses on the structure, dynamics, and function of neural circuits. For scientific questions addressing organisms operating autonomously in their environments, in particular where learning is involved, it is crucial to be able to operate such simulations in a closed-loop fashion. In such a set-up, the neural agent continuously receives sensory stimuli from the environment and provides motor signals that manipulate the environment or move the agent within it. So far, most studies requiring such functionality have been conducted with custom simulation scripts and manually implemented tasks. This makes it difficult for other researchers to reproduce and build upon previous work and nearly impossible to compare the performance of different learning architectures. In this work, we present a novel approach to solve this problem, connecting benchmark tools from the field of machine learning and state-of-the-art neural network simulators from computational neuroscience. The resulting toolchain enables researchers in both fields to make use of well-tested high-performance simulation software supporting biologically plausible neuron, synapse and network models and allows them to evaluate and compare their approach on the basis of standardized environments with various levels of complexity. We demonstrate the functionality of the toolchain by implementing a neuronal actor-critic architecture for reinforcement learning in the NEST simulator and successfully training it on two different environments from the OpenAI Gym. We compare its performance to a previously suggested neural network model of reinforcement learning in the basal ganglia and a generic Q-learning algorithm.

show abstract

RM-SORN: a reward-modulated self-organizing recurrent neural network

Cited by 13 publications

References 23 publications

Graph analysis of looming-selective networks in the tectum, and its replication in a simple computational model

Graph analysis of looming-selective networks in the tectum, and its replication in a simple computational model

Local Homeostatic Regulation of the Spectral Radius of Echo-State Networks

A Closed-Loop Toolchain for Neural Network Simulations of Learning Autonomous Agents

Contact Info

Product

Resources

About