Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Abstract

Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behavior. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of criticality. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos, hence producing unreliable computations. The performance gain due to optimizing hyper-parameters can be studied by considering the memory–nonlinearity trade-off, i.e., the fact that increasing the nonlinear behavior of the network degrades its ability to remember past inputs, and vice-versa. In this paper, we propose a model of ESNs that eliminates critical dependence on hyper-parameters, resulting in networks that provably cannot enter a chaotic regime and, at the same time, denotes nonlinear behavior in phase space characterized by a large memory of past inputs, comparable to the one of linear networks. Our contribution is supported by experiments corroborating our theoretical findings, showing that the proposed model displays dynamics that are rich-enough to approximate many common nonlinear systems used for benchmarking.

Physics-informed echo state networks

“…The commonlyused tanh activation offers good accuracy [13,18] for the systems studied here, as discussed in the results sections 3.1 and 3.2. While different activation functions have been proposed [19], it is beyond the scope of the present work to study the effect of activation functions on the echo state network accuracy. In the conventional ESN approach (Fig.…”

Performance Versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems

“…The ESN is a promising type of RNNs due to its relaxed training complexity and its ability to preserve the temporal features from different signals over time [21], [37]- [40]. The ESNs are in the reservoir computing (RC) category because in the ESNs only the output weights are trainable.…”

“…Finally, W in ∈ R Nr×ni is the input weight matrix, µ is the leaking rate parameter, and W o ∈ R no×Nr is the output weight matrix which is the only one that is trainable using a regression technique. This training phase in ESN does not affect the dynamics of the system, which makes it possible to operate with the same reservoir for different tasks [37]. A schematic representation of a leaky-ESN, including the sequential input, dynamic, static, and output layers, as depicted in Fig.…”

Input-to-State Representation in Linear Reservoirs Dynamics

“…Other hyper-parameters refer to the input and output scaling factors, the sparsity degree of W and the input bias. Moreover, the update equation ( 1) is usually chosen to be non-linear, i.e., x k = φ(W x k−1 + wu k ) and different choices of the nonlinear transfer function φ can be explored [42].…”