Weight statistics controls dynamics in recurrent neural networks

Abstract

Recurrent neural networks are complex non-linear systems, capable of ongoing activity in the absence of driving inputs. The dynamical properties of these systems, in particular their long-time attractor states, are determined on the microscopic level by the connection strengths w ij between the individual neurons. However, little is known to which extent network dynamics is tunable on a more coarse-grained level by the statistical features of the weight matrix. In this work, we investigate the dynamical impact of three statistical parameters: density (the fraction of non-zero connections), balance (the ratio of excitatory to inhibitory connections), and symmetry (the fraction of neuron pairs with w ij = w ji ). By computing a 'phase diagram' of network dynamics, we find that balance is the essential control parameter: Its gradual increase from negative to positive values drives the system from oscillatory behavior into a chaotic regime, and eventually into stationary fix points. Only directly at the border of the chaotic regime do the neural networks display rich but regular dynamics, thus enabling actual information processing. These results suggest that the brain, too, is fine-tuned to the 'edge of chaos' by assuring a proper balance between excitatory and inhibitory neural connections. Author summaryComputations in the brain need to be both reproducible and sensitive to changing input from the environment. It has been shown that recurrent neural networks can meet these simultaneous requirements only in a particular dynamical regime, called the edge of chaos in non-linear systems theory. Here, we demonstrate that recurrent neural networks can be easily tuned to this critical regime of optimal information processing by assuring a proper ratio of excitatory and inhibitory connections between the neurons. This result is in line with several micro-anatomical studies of the cortex, which frequently confirm that the excitatory-inhibitory balance is strictly conserved in the cortex. Furthermore, it turns out that neural dynamics is largely independent from the total density of connections, a feature that explains how the brain remains functional during periods of growth or decay. Finally, we find that the existence of too many symmetric connections is detrimental for the above mentioned critical dynamical regime, but maybe in turn useful for pattern completion tasks.

Analysis of continuous neuronal activity evoked by natural speech with computational corpus linguistics methods

“…for creating so-called embeddings of the raw data [122]. Moreover, as proposed by Kriegeskorte [123], our neural corpus can serve to test [124] computational models of brain function [125][126][127][128], in particular models based on neural networks [129][130][131] and machine learning architectures [132,133], in order to iteratively increase biological and cognitive fidelity [123].…”

Dynamical Phases and Resonance Phenomena in Information-Processing Recurrent Neural Networks

“…In this study, we investigated RNNs with random weight matrices, their ability to import and process information, and how both abilities depend on the density of non-zero weights and on the balance of excitatory and inhibitory connections, as introduced in previous studies [20,21].…”

“…It is therefore not very surprising that biological neural networks are also highly recurrent in their connectivity [16,17] making RNNs to versatile tools of neuroscience research [18,19]. Modelling natural RNNs in a realistic way requires the use of probabilistic, spiking neurons, but even simpler models with deterministic neurons already have highly complex dynamical properties and offer fascinating insights into how structure controls function in non-linear systems [20,21]. For example, we have demonstrated that by adjusting the density d of non-zero connections and the balance b between excitatory and inhibitory connections in the RNN's weight matrix, it is possible to control whether the system will predominantly end up in a periodic, chaotic, or fixed point attractor [21].…”

“…Modelling natural RNNs in a realistic way requires the use of probabilistic, spiking neurons, but even simpler models with deterministic neurons already have highly complex dynamical properties and offer fascinating insights into how structure controls function in non-linear systems [20,21]. For example, we have demonstrated that by adjusting the density d of non-zero connections and the balance b between excitatory and inhibitory connections in the RNN's weight matrix, it is possible to control whether the system will predominantly end up in a periodic, chaotic, or fixed point attractor [21]. Understanding and controlling the behavior of RNNs is of crucial importance for practical applications [22], especially as meaningful computation, or information processing, is believed to be only possible at the 'edge of chaos' [23][24][25][26][27][28][29].…”

Control of noise-induced coherent oscillations in three-neuron motifs

“…In information processing, networks take different tasks of functionality [37,38]. Thus, a better understanding of their structure and connectivity should shed more light on the dynamics of the phenomena occurring on them [39]. It is well-known that large recurrent networks can be decomposed into smaller building blocks -the so-called motifs [40], whereby threeneuron motifs (3NMs) are the most basic motifs, which frequently appear in neural circuits and can be seen as basic computational units [41], each uniquely contributing to a large-scale neural behavior [42,43].…”