A Practical Guide to Applying Echo State Networks

Abstract

Abstract. Reservoir computing has emerged in the last decade as an alternative to gradient descent methods for training recurrent neural networks. Echo State Network (ESN) is one of the key reservoir computing "flavors". While being practical, conceptually simple, and easy to implement, ESNs require some experience and insight to achieve the hailed good performance in many tasks. Here we present practical techniques and recommendations for successfully applying ESNs, as well as some more advanced application-specific modifications. IntroductionTraining Recurrent Neural Networks (RNNs) is inherently difficult. This (de-) motivates many to avoid them altogether. RNNs, however, represent a very powerful generic tool, integrating both large dynamical memory and highly adaptable computational capabilities. They are the Machine Learning (ML) model most closely resembling biological brains, the substrate of natural intelligence.Error backpropagation (BP) [40] is to this date one of the most important achievements in artificial neural network training. It has become the standard method to train especially Feed-Forward Neural Networks (FFNNs). Many useful practical aspects of BP are discussed in other chapters of this book and in its previous edition, e.g., [26]. BP methods have also been extended to RNNs [51,52], but only with a partial success. One of the conceptual limitations of BP methods for RNNs is that bifurcations can make training non-converging [8]. Even when they do converge, this convergence is slow, computationally expensive, and can lead to poor local minima.Ten years ago an alternative trend of understanding, training, and using RNNs has been proposed with Echo State Networks (ESNs) [16,21] in ML, and Liquid State Machines (LSMs) [32] in computational neuroscience. It was shown that RNNs often work well enough even without full adaptation of all network weights. In the classical ESN approach the RNN (called reservoir ) is generated randomly, and only the readout from the reservoir is trained. It should be noted that this basic idea was first clearly spelled out in a neuroscientific model of the corticostriatal processing loop [7]. Perhaps surprisingly this approach yielded excellent performance in many benchmark tasks, e.g., [16,15,19,22,47,48].

Investigations of the staircase memristor model and applications of memristor-based local connections

“…In order to evaluate the performance of the memristor based ESN with CNN structure (MCNN ESN), 10 running results are obtained for the proposed ESN with memristive CNN structure, the original ESN structure with memristive connections and the original ESN as shown in Table 2 using Python 2.7, Oger toolbox 1.1.3 and script developed by Mantas [21]. The results are measured using meansquared error (MSE) as shown in (25) which computes the differences between the predicted results and the test set of Mackey-Glass dataset.…”

“…However, in some trials, the readout weights of the memristive reservoir is obviously greater than the original ESN's for example in the range ðÀ6; 6Þ. According to the practical guide [21], large output weights W out may imply that the solution is sensitive and unstable because a tiny difference will be amplified by the output weights and lead to large deviations from the expected values. Therefore, the average performance of proposed ESN is slightly sensitive than the original ESN.…”

“…For the purpose of comparison, the original ESN is generated using the Python code developed by Mantas [21].…”

Noise robust continuous digit recognition with reservoir-based acoustic models

“…This is achieved by means of a random process characterized by four control parameters (see [14] for more details). These parameters are: (1) α U , the maximal absolute eigenvalue of the input weight matrix W in , (2) ρ, the maximal absolute eigenvalue of the recurrent weight matrix W rec , (3) K in , the number of inputs driving each reservoir neuron and (4) K rec , the number of delayed reservoir outputs driving each reservoir neuron.…”

“…= −τ f r / ln(1 − λ) as the leaky integration time constant and T as the expected state duration, we select (ρ, τ λ , K in , K rec ) = (0.8, T, 10, 10). The parameter α U is chosen so that the average variance of the resrevoir outputs reach a certain level [14]. Since layers 2 and 3 see basically the same inputs, α U is taken the same for both layers.…”