Echo State Networks with Sparse Output Connections

Kobialka, Hans-Ulrich; Kayani, Umer

doi:10.1007/978-3-642-15819-3_47

Cited by 12 publications

(12 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They investigate different greedy methods to this end, including backward selection (where connections are removed one at a time based on an iterative procedure), random deletion, and others. Similar experiments are conducted by Kobialka and Kayani [40].…”

Section: B Sparse Readouts For Esnssupporting

confidence: 52%

Distributed Reservoir Computing with Sparse Readouts [Research Frontier]

Scardapane

Panella

Comminiello

et al. 2016

IEEE Comput. Intell. Mag.

View full text Add to dashboard Cite

In a network of agents, a widespread problem is the need to estimate a common underlying function starting from locally distributed measurements. Real-world scenarios may not allow the presence of centralized fusion centers, requiring the development of distributed, message-passing implementations of the standard machine learning training algorithms. In this paper, we are concerned with the distributed training of a particular class of recurrent neural networks, namely echo state networks (ESNs). In the centralized case, ESNs have received considerable attention, due to the fact that they can be trained with standard linear regression routines. Based on this observation, in our previous work we have introduced a decentralized algorithm, framed in the distributed optimization field, in order to train an ESN. In this paper, we focus on an additional sparsity property of the output layer of ESNs, allowing for very efficient implementations of the resulting networks. In order to evaluate the proposed algorithm, we test it on two well-known prediction benchmarks, namely the Mackey-Glass chaotic time series and the 10th order nonlinear auto regressive moving average (NARMA) system.

show abstract

Section: B Sparse Readouts For Esnssupporting

confidence: 52%

Distributed Reservoir Computing with Sparse Readouts [Research Frontier]

Scardapane

Panella

Comminiello

et al. 2016

IEEE Comput. Intell. Mag.

View full text Add to dashboard Cite

show abstract

“…The usual benefits of using sparse regression algorithms are efficient suppression of noise, easier interpretation of the results, improved generalisation properties, reduced surplus of inputs (in this case readouts) and other [7].…”

Section: Sparse Linear Regressionmentioning

confidence: 99%

“…In [7] it is shown the use of feature selection algorithms (backward elimination and Markov blanket-embedded genetic algorithm (MBEGA)) for reduction of the number of required readouts can reduce the generalisation error of the ESN, i.e. it is desirable to reduce the overall number of readouts.…”

Section: Introductionmentioning

confidence: 99%

Reducing Complexity of Echo State Networks with Sparse Linear Regression Algorithms

Čeperić

Barić

2014

2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation

View full text Add to dashboard Cite

In this paper the use of sparse linear regression algorithms in echo state networks (ESN) is presented for reducing the number of readouts and improving the robustness and generalization properties of ESNs. Three data sets with overall 80 tests are used to validate the use of sparse linear regression algorithms for echo state networks. It is shown that it is possible to increase accuracy on the test data sets, not used in the ESN training phase, and in the same time reduce the overall number of the required readouts when compared to the standard approach of using ridge linear regression on the echo state network readouts.

show abstract

“…Not all dimensions may contribute to the solution. The internal layer of ESN is sparsely connected, hence, the fact that each output node is connected to all internal nodes seems contradictory [20]. Therefore, the output connection of ESN should be optimized.…”

Section: Introductionmentioning

confidence: 98%

“…[28]. Kobialka and Kayani use a greedy feature selection algorithm to exclude irrelevant internal ESN states [20]. The optimization of the connection structure of output weights is a problem of whether internal and output layer nodes are connected, thus, it is a discrete optimization problem.…”

Section: Introductionmentioning

confidence: 99%