A Characterization of the Edge of Criticality in Binary Echo State Networks

Abstract

Echo State Newtworks (ESNs) are simplified recurrent neural network models composed of a reservoir and a linear, trainable readout layer. The reservoir is tunable by some hyper-parameters that control the network behaviour. ESNs are known to be effective in solving tasks when configured on a region in (hyper-)parameter space called Edge of Criticality (EoC), where the system is maximally sensitive to perturbations hence affecting its behaviour. In this paper, we propose binary ESNs, which are architecturally equivalent to standard ESNs but consider binary activation functions and binary recurrent weights. For these networks, we derive a closed-form expression for the EoC in the autonomous case and perform simulations in order to assess their behavior in the case of noisy neurons and in the presence of a signal. We propose a theoretical explanation for the fact that the variance of the input plays a major role in characterizing the EoC.

Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

“…Fine tuning of hyper-parameters requires cross-validation or ad-hoc criteria for selecting the best-performing configuration. Experimental evidence and some results from the theory show that ESNs performance is usually maximized in correspondence of a very narrow region in hyper-parameter space called Edge of Criticality (EoC) [13][14][15][16][17][18][19][20] . However, we comment that beyond such a region ESNs behave chaotically, resulting in useless and unreliable computations.…”

Input-to-State Representation in Linear Reservoirs Dynamics

“…In particular, it appears that the hyper-parameter space can be divided into a region where the dynamics are "regular" (meaning that they are stable with respect to the inputs driving the system) and another one where they are "disordered" (meaning that they are unstable and do not provide a representation for the input) [43]. The narrow region separating these two regimes is known in the literature as Edge of Chaos or Edge of Criticality (EoC) [44,45,46] and appears to be common to a large variety of complex systems beyond RNNs [47,48,49].…”