Information processing in echo state networks at the edge of chaos

Abstract

We investigate information processing in randomly connected recurrent neural networks. It has been shown previously that the computational capabilities of these networks are maximized when the recurrent layer is close to the border between a stable and an unstable dynamics regime, the so called edge of chaos. The reasons, however, for this maximized performance are not completely understood. We adopt an information-theoretical framework and are for the first time able to quantify the computational capabilities between elements of these networks directly as they undergo the phase transition to chaos. Specifically, we present evidence that both information transfer and storage in the recurrent layer are maximized close to this phase transition, providing an explanation for why guiding the recurrent layer toward the edge of chaos is computationally useful. As a consequence, our study suggests self-organized ways of improving performance in recurrent neural networks, driven by input data. Moreover, the networks we study share important features with biological systems such as feedback connections and online computation on input streams. A key example is the cerebral cortex, which was shown to also operate close to the edge of chaos. Consequently, the behavior of model systems as studied here is likely to shed light on reasons why biological systems are tuned into this specific regime.

Criticality and Information Dynamics in Epidemiological Models

“…(Indeed, as noted earlier, SIS dynamics have been used to model dynamics on brain networks [38]). Our results in this paper regarding SIS dynamics continue to add to the quantitative evidence regarding the maximisation of information storage and transfer as intrinsic computational properties at or near critical states, as previously found in a diverse range of dynamics and network structures including the Ising model [25], recurrent neural networks [26], gene regulatory network (GRN) models [23], and regular-small-world-random transitions in structure [24]. In this way, we have provided another important link for epidemic spreading models to complex networks, criticality and information dynamics.…”

“…This is particularly the case in complex systems and network approaches to computational neuroscience, where it is conjectured that the brain is in or near a critical state so as to advantageously use maximised computational properties here [26,[45][46][47][48]. (Indeed, as noted earlier, SIS dynamics have been used to model dynamics on brain networks [38]).…”

“…Information transfer, from a collection of sources, has also been measured to peak prior to the phase transition in the Ising model, when the critical point is approached from the disordered (high-temperature) side [25]. And similarly, both information storage and transfer were measured to be maximised near the critical state in echo-state networks (a type of recurrent neural network) [26], where such networks are argued to be best placed for general-purpose computation.…”

Echo State Condition at the Critical Point

“…One can try |λ| max = 1 and see if this improves the network performance and how this impacts the memory of the reservoir on earlier events. Steps toward this direction have been made by going near the "edge of chaos" [5] or even further where the network may not be an echo state network for all possible input sequences but instead just around some permissible inputs [6]. Presumably, these approaches still all forget exponentially fast.…”

“…Although a random recurrent connectivity pattern can be used, heuristically it has been found that typically the performance of the network depends strongly on the statistical features of this random connectivity (cf. for example [5] …”