Reducing Complexity of Echo State Networks with Sparse Linear Regression Algorithms

Čeperić, Vladimir; Barić, Adrijan

doi:10.1109/uksim.2014.36

Cited by 8 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the training of the readout is formulated as a linear problem, most of the relevant literature can also be applied for this class of networks. As an example, ℓ 1 minimization to achieve sparse readouts was investigated independently in the context of ESNs by Ceperic and Baric and Bianchi et al, with some similar results obtained in a previous study by Butcher et al Additionally, it is possible to consider advanced optimization strategies, support vector algorithms, and many other variations (see Section 7 of Lukoševičius and Jaeger for an overview).…”

Section: Theoretical Properties Of Rc Networkmentioning

confidence: 74%

“…At the same time, the literature on these topics is vast and fragmented so that equivalent ideas are reintroduced time and again, and it becomes difficult to appreciate the fundamental unity (both theoretical and practical) underlying all these methods. Going back to our previous example, sparse linear regression has been derived independently in all three areas, sometimes more than once in each case …”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Randomness in neural networks: an overview

Scardapane

Wang

2017

WIREs Data Min & Knowl

247

144

View full text Add to dashboard Cite

Neural networks, as powerful tools for data mining and knowledge engineering, can learn from data to build feature-based classifiers and nonlinear predictive models. Training neural networks involves the optimization of nonconvex objective functions, and usually, the learning process is costly and infeasible for applications associated with data streams. A possible, albeit counterintuitive, alternative is to randomly assign a subset of the networks' weights so that the resulting optimization task can be formulated as a linear least-squares problem. This methodology can be applied to both feedforward and recurrent networks, and similar techniques can be used to approximate kernel functions. Many experimental results indicate that such randomized models can reach sound performance compared to fully adaptable ones, with a number of favorable benefits, including (1) simplicity of implementation, (2) faster learning with less intervention from human beings, and (3) possibility of leveraging overall linear regression and classification algorithms (e.g., ℓ 1 norm minimization for obtaining sparse formulations). This class of neural networks attractive and valuable to the data mining community, particularly for handling large scale data mining in real-time. However, the literature in the field is extremely vast and fragmented, with many results being reintroduced multiple times under different names. This overview aims to provide a self-contained, uniform introduction to the different ways in which randomization can be applied to the design of neural networks and kernel functions. A clear exposition of the basic framework underlying all these approaches helps to clarify innovative lines of research, open problems, and most importantly, foster the exchanges of well-known results throughout different communities.

show abstract

Section: Theoretical Properties Of Rc Networkmentioning

confidence: 74%

Section: Introductionmentioning

confidence: 99%

Randomness in neural networks: an overview

Scardapane

Wang

2017

WIREs Data Min & Knowl

247

144

View full text Add to dashboard Cite

show abstract

“…where L is the length of training data, y i is the training label that will be discussed in the next section and λ represents the regularisation parameter that determines the strength of the L 1 penalty [36,37]. Finally, the output X can be written as:…”

Section: Lasso Regressionmentioning

confidence: 99%

A Neuromorphic Model With Delay-Based Reservoir for Continuous Ventricular Heartbeat Detection

Liang¹,

Li²,

Vučković³

et al. 2022

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

“…The use of the LASSO algorithm, where sparsity is obtained by including an additional L 1 regularization term, is derived by Ceperic and Baric [41]. It is also possible to combine ridge regression with the LASSO algorithm, obtaining the so-called elastic net penalty [42].…”

Section: B Sparse Readouts For Esnsmentioning

confidence: 99%

“…It is also possible to combine ridge regression with the LASSO algorithm, obtaining the so-called elastic net penalty [42]. This has been investigated independently by Ceperic and Baric [41], and Bianchi et al [22].…”

Section: B Sparse Readouts For Esnsmentioning

confidence: 99%

Distributed Reservoir Computing with Sparse Readouts [Research Frontier]

Scardapane

Panella

Comminiello

et al. 2016

IEEE Comput. Intell. Mag.

View full text Add to dashboard Cite

In a network of agents, a widespread problem is the need to estimate a common underlying function starting from locally distributed measurements. Real-world scenarios may not allow the presence of centralized fusion centers, requiring the development of distributed, message-passing implementations of the standard machine learning training algorithms. In this paper, we are concerned with the distributed training of a particular class of recurrent neural networks, namely echo state networks (ESNs). In the centralized case, ESNs have received considerable attention, due to the fact that they can be trained with standard linear regression routines. Based on this observation, in our previous work we have introduced a decentralized algorithm, framed in the distributed optimization field, in order to train an ESN. In this paper, we focus on an additional sparsity property of the output layer of ESNs, allowing for very efficient implementations of the resulting networks. In order to evaluate the proposed algorithm, we test it on two well-known prediction benchmarks, namely the Mackey-Glass chaotic time series and the 10th order nonlinear auto regressive moving average (NARMA) system.

show abstract

Reducing Complexity of Echo State Networks with Sparse Linear Regression Algorithms

Cited by 8 publications

References 15 publications

Randomness in neural networks: an overview

Randomness in neural networks: an overview

A Neuromorphic Model With Delay-Based Reservoir for Continuous Ventricular Heartbeat Detection

Distributed Reservoir Computing with Sparse Readouts [Research Frontier]

Contact Info

Product

Resources

About