Computational Efficiency of Multi-Step Learning Echo State Networks for Nonlinear Time Series Prediction

Akiyama, Takanori; Tanaka, Gouhei

doi:10.1109/access.2022.3158755

Cited by 15 publications

(15 citation statements)

References 26 publications

(40 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We show the results in performance in Table 1. These test scores are in line with our previous observations, as both DDNs perform consistently better than their respective baselines, but also similarly or better compared what is reported in several recent novel ESN implementations [5,17,1].…”

Section: Resultssupporting

confidence: 91%

Distance-Based Delays in Echo State Networks

Iacob

Freiberger²,

Dambre³

2022

Intelligent Data Engineering and Automated Learning – IDEAL 2022

View full text Add to dashboard Cite

Physical reservoir computing, a paradigm bearing the promise of energy-efficient high-performance computing, has raised much attention in recent years. We argue though, that the effect of signal propagation delay on reservoir task performance, one of the most central aspects of physical reservoirs, is still insufficiently understood in a more general learning context. Such physically imposed delay has been found to play a crucial role in some specific physical realizations, such as integrated photonic reservoirs. While delays at the readout layer and input of Echo State Networks (ESNs) have been successfully exploited before to improve performance, to our knowledge this feature has not been studied in a more general setting. We introduce inter-node delays, based on physical distances, into ESNs as model systems for physical reservoir computing. We propose a novel ESN design that includes variable signal delays along the connections between neurons, comparable to varying axon lengths in biological neural networks or varying length delay lines in physical systems. We study the impact of the resulting variable inter-node delays in this setup in comparison with conventional ESNs and find that incorporating variable delays significantly improves reservoir performance on the NARMA-10 benchmark task.

show abstract

Section: Resultssupporting

confidence: 91%

Distance-Based Delays in Echo State Networks

Iacob

Freiberger²,

Dambre³

2022

Intelligent Data Engineering and Automated Learning – IDEAL 2022

View full text Add to dashboard Cite

show abstract

“…This is crucial for understanding the temporal dynamics in time series data samples. The self-attention mechanism is formalized via equation 10, 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄, 𝐾, 𝑉) = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 ( 𝑄𝐾 𝑇 𝒅𝒌 ) 𝑉 … (10) Where, Q, K, and V represent the queries, keys, and values matrices, respectively, derived from the input data, and dk is the dimensionality of the keys. This equation ensures that each output element is a weighted sum of the values, with weights computed based on the input's relevance.…”

confidence: 99%

Design of an Iterative Method for Enhanced Multimodal Time Series Analysis Using Graph Attention Networks, Variational Graph Autoencoders, and Transfer Learning

Vijaya Kamble

2024

jes

View full text Add to dashboard Cite

In the ever-evolving landscape of data analysis, the need to efficiently and accurately interpret multimodal time series data has become paramount. Traditional methods often fall short in addressing the complex dependencies and dynamics inherent in such data, limiting their effectiveness in real-world applications. This work introduces a comprehensive approach that leverages Graph Attention Networks (GATs), Variational Graph Autoencoders (VGAEs), transfer learning with pretrained transformers, and Bayesian state-space models to overcome these limitations. GATs are selected for their ability to dynamically focus on relevant modalities through attention mechanisms, thereby capturing the intricate relationships between different data modalities. This method significantly enhances the model's ability to integrate multimodal information, leading to notable improvements in classification, prediction, and anomaly detection tasks. VGAEs are utilized to learn latent representations within a graph-based framework, promoting unsupervised learning while unveiling the underlying data structure. The resultant embeddings are pivotal for downstream tasks like clustering and visualization, encapsulating the interactions within multimodal time series data effectively. Furthermore, this work incorporates transfer learning with pretrained transformers to harness extensive knowledge from large datasets, adapting it to multimodal time series analysis. This strategy excels in capturing long-range dependencies, thereby augmenting generalization and performance in data-scarce scenarios. Bayesian state-space models are employed to elucidate the temporal dynamics and uncertainties of time series data, offering a robust framework for probabilistic inference and enhancing the interpretability and reliability of forecasting and anomaly detection. The efficacy of the proposed model is rigorously evaluated using diverse datasets, including the Yahoo! Stock Dataset, Forest Cover Dataset, and an empirical collection of 100k time series data samples. The results demonstrate a significant leap in performance metrics, including a 9.5% increase in precision, 8.5% boost in accuracy, 8.3% rise in recall, 10.4% reduction in delay, 9.4% enhancement in AUC, and a 5.9% improvement in specificity, alongside superior pre-emption capabilities compared to existing methods. This work not only addresses the pressing need for advanced multimodal time series analysis techniques but also sets a new benchmark for efficiency and accuracy. The integration of GATs, VGAEs, transfer learning with pretrained transformers, and Bayesian state-space models presents a formidable approach that significantly advances the field, offering profound impacts on a wide array of applications.

show abstract

“…Then, in the reservoir layer, input time series data is transformed nonlinearly to high dimensional space as reservoir states X i (k) at given node i (i = 1, 2, …, N). 1,31) For the output layer, the readout weight w i connecting X i (k) and output y i (k) is trained by linear regression to obtain the desired output. The reservoir output y(k) is described as a linear combination of X i (k) and the w i , as follows,…”

mentioning

confidence: 99%

“…The function of the reservoir in reservoir computing is to map the input to a high-dimensional feature space. 1,31) The reservoir's mapping function f i for the input u i and reservoir state X i is shown in Fig. 1(a).…”

mentioning

confidence: 99%

“…In a simulation reservoir such as an echo state network, nonlinear functions such as tanh are employed identically for all nodes (i.e., f i = f j≠i ). 25,31) On the other hand, in PRC, a variety of nonlinear functions that originate from nonlinear dynamics that are inherent in the physical system are employed as mapping functions for each node (i.e., for each measurement terminal or sampling time/virtual node). 1, These mapping functions change dynamically according to the driving conditions (e.g., input intensity and time scale) of the physical system (i.e., f i ≠ f j≠i ).…”

mentioning

confidence: 99%

See 1 more Smart Citation

Inverted input method for computing performance enhancement of the ion-gating reservoir

Yamaguchi,

Nishioka,

Namiki

et al. 2024

Appl. Phys. Express

View full text Add to dashboard Cite

Physical reservoir computing (PRC) is useful for edge computing, although the challenge is to improve computational performance. In this study, we developed an inverted input method, that the inverted input is additionally applied to a physical reservoir together with the original input, to improve the performance of the ion-gating reservoir. The error in a second-order nonlinear equation task was 7.3×10-5, the lowest error in reported PRC to date. Improvement of high dimensionality by the method was confirmed to be the origin of the performance enhancement. This inverted input method is versatile enough to enhance the performance of any other PRC.

show abstract

Computational Efficiency of Multi-Step Learning Echo State Networks for Nonlinear Time Series Prediction

Cited by 15 publications

References 26 publications

Distance-Based Delays in Echo State Networks

Distance-Based Delays in Echo State Networks

Design of an Iterative Method for Enhanced Multimodal Time Series Analysis Using Graph Attention Networks, Variational Graph Autoencoders, and Transfer Learning

Inverted input method for computing performance enhancement of the ion-gating reservoir

Contact Info

Product

Resources

About