A Gated Recurrent Unit based Echo State Network

Wang, Xinjie; Jin, Yaochu; Hao, Kuangrong

doi:10.1109/ijcnn48605.2020.9206786

Cited by 4 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ReservoirComputing.jl provides a simple interface for model building that closely follows the workflow presented in the corresponding literature. It includes a standard implementation of ESNs [Lukoševičius, 2012] as well as a hybrid variation [Pathak et al, 2018b], gated recurrent unit ESN [Wang et al, 2020] and double activation function ESN [Lun et al, 2015]. Multiple input layers χ and reservoirs R are also provided, ranging from weighted input layers [Lu et al, 2017] to minimally complex input layers and reservoirs Tino, 2010, Rodan andTiňo, 2012], including a reservoir obtained through pseudo single value decomposition [Yang et al, 2018].…”

Section: Library Overviewmentioning

confidence: 99%

ReservoirComputing.jl: An Efficient and Modular Library for Reservoir Computing Models

Martinuzzi¹,

Rackauckas²,

Abdelrehim³

et al. 2022

Preprint

View full text Add to dashboard Cite

We introduce ReservoirComputing.jl, an open source Julia library for reservoir computing models. The software offers a great number of algorithms presented in the literature, and allows to expand on them with both internal and external tools in a simple way. The implementation is highly modular, fast and comes with a comprehensive documentation, which includes reproduced experiments from literature. The code and documentation are hosted on Github under an MIT license https:// github.com/SciML/ReservoirComputing.jl.

show abstract

Section: Library Overviewmentioning

confidence: 99%

ReservoirComputing.jl: An Efficient and Modular Library for Reservoir Computing Models

Martinuzzi¹,

Rackauckas²,

Abdelrehim³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The process of extending the architecture of ESNs with gating mechanisms has been first investigated in our previous work [9] and, concurrently and independently, in [32]. The two works share a similar idea, which consists of extending the state transition function of an ESN to include the same gating mechanisms of a GRU, while keeping all the parameters in the gates untrained.…”

Section: Related Workmentioning

confidence: 99%

“…While the underlying idea from [9] and [32] is similar, in [9] and in the current work we also take care to (1) consider the fundamental details regarding the initialization strategy, (2) perform an experimental evaluation over a dataset that makes it possible to evaluate more clearly the actual influence of the gates, and (3) perform a more extensive evaluation of the competing reservoir computing models, which led to important insights about the feasibility of the approach. Moreover, the current paper further extends our previous work [9] to include (a) the development of a theoretical analysis of the state dynamics in the proposed models, (b) the analysis and discussion of the agreement between the theoretical results and the experimental measurements, (c) the expansion of the hyperparameter search for the experiments, (d) the collection and discussion of additional measurements for the activation of the gates, and (e) the reporting and discussion of additional measurements regarding the weight matrices.…”

Section: Related Workmentioning

confidence: 99%

On the effectiveness of Gated Echo State Networks for data exhibiting long-term dependencies

2022

View full text Add to dashboard Cite

In the context of recurrent neural networks, gated architectures such as the GRU have contributed to the development of highly accurate machine learning models that can tackle long-term dependencies in the data. However, the training of such networks is performed by the expensive algorithm of gradient descent with backpropagation through time. On the other hand, reservoir computing approaches such as Echo State Networks (ESNs) can produce models that can be trained efficiently thanks to the use of fixed random parameters, but are not ideal for dealing with data presenting long-term dependencies. We explore the problem of employing gated architectures in ESNs from both theoretical and empirical perspectives. We do so by deriving and evaluating a necessary condition for the non-contractivity of the state transition function, which is important to overcome the fading-memory characterization of conventional ESNs. We find that using pure reservoir computing methodologies is not sufficient for effective gating mechanisms, while instead training even only the gates is highly effective in terms of predictive accuracy.

show abstract

“…Gated recurrent unit (GRU) [133] is another enhanced version of RNN. In [134,135], authors presented GRU-based ESNs to tackle complex real-world tasks while reducing the computational costs. The reservoir unit was replaced by the sparsely connected GRU neurons.…”

Section: Combinations Of Esn and Deep Learningmentioning

confidence: 99%

“…To test ESN, most existing works set s = 5 and α 1 = 0.2, α 2 = 0.311, α 3 = 0.42, α 4 = 0.51, α 5 = 0.63. [43,134] • Lorenz system.…”

Section: Benchmark Datasetsmentioning

confidence: 99%

A Review of Designs and Applications of Echo State Networks

Sun¹,

Song²,

Hong³

et al. 2020

Preprint

View full text Add to dashboard Cite

Recurrent Neural Networks (RNNs) have demonstrated their outstanding ability in sequence tasks and have achieved state-of-the-art in wide range of applications, such as industrial, medical, economic and linguistic. Echo State Network (ESN) is simple type of RNNs and has emerged in the last decade as an alternative to gradient descent training based RNNs. ESN, with a strong theoretical ground, is practical, conceptually simple, easy to implement. It avoids non-converging and computationally expensive in the gradient descent methods. Since ESN was put forward in 2002, abundant existing works have promoted the progress of ESN, and the recently introduced Deep ESN model opened the way to uniting the merits of deep learning and ESNs. Besides, the combinations of ESNs with other machine learning models have also overperformed baselines in some applications. However, the apparent simplicity of ESNs can sometimes be deceptive and successfully applying ESNs needs some experience. Thus, in this paper, we categorize the ESN-based methods to basic ESNs, DeepESNs and combinations, then analyze them from the perspective of theoretical studies, network designs and specific applications. Finally, we discuss the challenges and opportunities of ESNs by summarizing the open questions and proposing possible future works.

show abstract

A Gated Recurrent Unit based Echo State Network

Cited by 4 publications

References 23 publications

ReservoirComputing.jl: An Efficient and Modular Library for Reservoir Computing Models

ReservoirComputing.jl: An Efficient and Modular Library for Reservoir Computing Models

On the effectiveness of Gated Echo State Networks for data exhibiting long-term dependencies

A Review of Designs and Applications of Echo State Networks

Contact Info

Product

Resources

About