Design of deep echo state networks

Abstract

In this paper, we provide a novel approach to the architectural design of deep Recurrent Neural Networks using signal frequency analysis. In particular, focusing on the Reservoir Computing framework and inspired by the principles related to the inherent effect of layering, we address a fundamental open issue in deep learning, namely the question of how to establish the number of layers in recurrent architectures in the form of deep echo state networks (DeepESNs). The proposed method is first analyzed and refined on a controlled scenario and then it is experimentally assessed on challenging real-world tasks. The achieved results also show the ability of properly designed DeepESNs to outperform RC approaches on a speech recognition task, and to compete with the state-of-the-art in time-series prediction on polyphonic music tasks.

Reservoir Topology in Deep Echo State Networks

“…On the one hand, the analysis of DeepESN dynamics contributes to uncover the intrinsic computational properties of deep neural networks in the temporal domain [7,12]. On the other hand, a proper architectural design of deep reservoirs might have a huge impact in real-world applications [11], enabling effective multiple time-scales processing and at the same time preserving the training efficiency typical of RC models.…”

Analysis of Wide and Deep Echo State Networks for Multiscale Spatiotemporal Time Series Forecasting

“…To this end, various architectures have been proposed with multiple reservoirs, additional projections, autoencoders, plasticity mechanisms, etc. [5,6,12,14,29,32].…”

“…The more straightforward way of measuring reservoir goodness is by evaluating the performance of a network on a task. For scalarvalued time series forecasting tasks, we consider the following three metrics to quantify network error: root-mean-square error (RMSE) (14), normalized RMSE (NRMSE) (15), and mean absolute percentage error (MAPE) (16).…”

“…Prior literature has shown that ESNs are capable of various functions, such as speech processing, EEG classification, and anomaly detection [24,44]. In recent literature, several groups have begun to study how these networks can cope with increasingly complex time series tasks with dynamics across multiple scales and domains [5,14,29,32]. One technique to enhance ESNs is the addition of reservoir layers.…”

Richness of Deep Echo State Network Dynamics

“…A promising research line in the current development of RC is given by the exploration of its extensions to deep learning [16,22], with the introduction of Deep Echo State Network (DeepESN) [5] providing a refreshing perspective to the study of hierarchically structured RNNs. On the one hand, results in [10,6] suggested that a proper architectural design of DeepESNs can have a tremendous impact on real-world applications. On the other hand, investigations on DeepESNs dynamics [5,4,11] revealed that a stacked composition of recurrent layers has the inherent ability to diversify the dynamical response to the driving input signals in successive levels of the architecture.…”