Cluster-Based Input Weight Initialization for Echo State Networks

Abstract

Echo state networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image, and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the K -means algorithm on the training data. We show that for a large variety of datasets, this initialization performs equivalently or superior than a randomly initialized ESN while needing significantly less reservoir neurons. Furthermore, we discuss that this approach provides the opportunity to estimate a suitable size of the reservoir based on prior knowledge about the data.

PyRCN: A toolbox for exploration and application of Reservoir Computing Networks

“…We have demonstrated how to use the toolbox on two widely known tasks, namely time-series prediction with the Mackey-Glass equation and handwritten digit classification using the handwritten digits dataset. Based on a benchmark test, we have shown that we can reproduce the results by Trierweiler Ribeiro et al (2021) using PyRCN with fewer lines of code, and that the sequential optimization scheme proposed and used by Steiner et al (2021Steiner et al ( , 2020Steiner et al ( , 2022 is applicable to univariate time-series. By comparing PyRCN with reference toolboxes, we have shown that the required computational time of PyRCN is comparable in case of small reservoirs and outperforms the reference toolboxes by decades in case of large reservoirs as PyRCN was in the mean ten times faster than PyESN.…”

“…Interestingly, by limiting the optimization of the PyESN toolbox to the aforementioned hyper-parameters, we slightly outperformed Trierweiler Ribeiro et al ( 2021) in many cases. This suggests that our proposed way of sequential optimization from Steiner et al (2021Steiner et al ( , 2022 is also applicable for this kind of tasks.…”

“…This is, for example, necessary if the input weights should be pre-trained e.g. by Batch Intrinsic Plasticity (BIP) (Neumann and Steil, 2011) or other unsupervised pre-training methods, such as in Steiner et al (2022). An example for an ELM with a BIP ''Input-to-Node'' and a specific ''Node-to-Output'' is given in Listing 6.…”

“…As for ELMs in Listing 6, various unsupervised learning techniques can be used to pre-train ESNs (Basterrech et al, 2011;Schrauwen et al, 2008;Lazar et al, 2009). For a summary, we refer to Steiner et al (2022). An example is given in Listing 9, in which we use a BIP ''Input-to-Node'' and a Hebbian ''Node-to-Node''.…”

Unsupervised Pretraining of Echo State Networks for Onset Detection

“…Recently, in [14], we have shown that ESNs strongly benefit from a clusterbased input weight initialization, leading to equivalent or superior results compared to a baseline ESN whilst needing significantly less reservoir neurons, which consequently reduces the required amount of free parameters and training time.…”