Dynamics and Information Import in Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) are complex dynamical systems, capable of ongoing activity without any driving input. The long-term behavior of free-running RNNs, described by periodic, chaotic and fixed point attractors, is controlled by the statistics of the neural connection weights, such as the density d of non-zero connections, or the balance b between excitatory and inhibitory connections. However, for information processing purposes, RNNs need to receive external input signals, and it is not clear which of the dynamical regimes is optimal for this information import. We use both the average correlations C and the mutual information I between the momentary input vector and the next system state vector as quantitative measures of information import and analyze their dependence on the balance and density of the network. Remarkably, both resulting phase diagrams C(b, d) and I(b, d) are highly consistent, pointing to a link between the dynamical systems and the information-processing approach to complex systems. Information import is maximal not at the “edge of chaos,” which is optimally suited for computation, but surprisingly in the low-density chaotic regime and at the border between the chaotic and fixed point regime. Moreover, we find a completely new type of resonance phenomenon, which we call “Import Resonance” (IR), where the information import shows a maximum, i.e., a peak-like dependence on the coupling strength between the RNN and its external input. IR complements previously found Recurrence Resonance (RR), where correlation and mutual information of successive system states peak for a certain amplitude of noise added to the system. Both IR and RR can be exploited to optimize information processing in artificial neural networks and might also play a crucial role in biological neural systems.

Intrinsic Noise Improves Speech Recognition in a Computational Model of the Auditory Pathway

“…Our study provides evidence that an interplay of deep learning and neuroscience helps on the one hand to raise understanding of the function of biological neural networks and cognition in general (e.g., Schilling et al, 2018 , 2021b ; Krauss et al, 2019a , c , d , 2021 ; Gerum et al, 2020 ; Krauss and Maier, 2020 ; Bönsel et al, 2021 ; Metzner and Krauss, 2022 ), an emerging science strand referred to as cognitive computational neuroscience ( Kriegeskorte and Douglas, 2018 ). On the other hand, fundamental processing principles from nature—such as stochastic resonance—can be transferred to improve artificial neural systems, which is called neuroscience-inspired AI ( Hassabis et al, 2017 ; Gerum et al, 2020 ; Gerum and Schilling, 2021 ; Yang et al, 2021 ; Maier et al, 2022 ).…”

Timescales: the choreography of classical and unconventional computing

“…Unfortunately, numerically estimating the mutual information from samples of two distributions is only approximate, computationally expensive and becomes practical only when one makes additional assumptions about the distributions. In practice one will often take resort to correlational measures which sometimes provide qualitatively similar insights as mutual information (Metzner & Krauss, 2021).…”

Classification at the accuracy limit: facing the problem of data ambiguity

“…By color-coding each projected data point of a data set according to its label, the representation of the data can be visualized as a set of point clusters. For instance, MDS has already been applied to visualize for instance word class distributions of different linguistic corpora 23 , hidden layer representations (embeddings) of artificial neural networks 6,15,24 , structure and dynamics of recurrent neural networks [25][26][27][28] , or brain activity patterns assessed during e.g. pure tone or speech perception 14,23,29 , or even during sleep 5,30 .…”