Multi-step forecasting using echo state networks

Abstract

Echo State Networks (ESNs) have been recently proposed as a special class of recurrent neural networks (RNNs), which help to avoid the possibility of vanishing gradient associated with RNNs, and also computational less complex. On-line training of ESNs has previously been implemented using an RLS-type algorithm. Our approach aims at avoiding the numerical disadvantages inherent to the RLS algorithm by switching to a simpler and less computationally-intensive gradient descent algorithm. Simulations performed on benchmark AR, nonlinear and chaotic signals suggest that the performance of ESNs in single-step and multistep-ahead prediction is not sacrificed by the proposed method.

Spatio-Temporal Learning With the Online Finite and Infinite Echo-State Gaussian Processes

“…For online training, this regression can be performed as the reservoir evolves over time via stochastic gradient descent (SGD) as demonstrated in [24]; the output weights are updated iteratively,…”

“…It was shown the SGD-ESN was sufficient to perform multi-step tracking of a 3-D Lorenz system [24]. That said, convergence performance of the SGD approach is not only heavily dependent on η but is also negatively impacted by the eigenvalue spread of the reservoir cross-correlation matrix [20].…”

Iterative temporal learning and prediction with the sparse online echo state gaussian process

“…As a cheaper O(N ψ ) alternative, Kountoriotis et al [10] proposed updating the output weights directly via stochastic gradient descent (SGD) or equivalently, Least Means Squares (LMS), i.e. :…”

“…Unlike current online ESNs trained using recursive least squares (RLS) [9] and stochastic gradient descent (SGD) [10], our Bayesian-based formulation gives predictive distributions instead of point-predictions. One can view the OESGP as an online variant of the recently-proposed echo state gaussian process (ESGP) [11], a combination of the ESN and gaussian process (GP) approaches.…”

“…This section provides the necessary background for our work; we first give an overview of ESNs, with a focus on online variants proposed by Jaeger [9] and Kountouriotis et al [10], followed by a brief review of the ESGP [11].…”