Limitations of the Recall Capabilities in Delay-Based Reservoir Computing Systems

Köster, Felix; Ehlert, Dominik; Lüdge, Kathy

doi:10.1007/s12559-020-09733-5

Cited by 27 publications

(36 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The inclusion of delayed input significantly reduced the NARMA10 error, reaching an NRMSE of about 0.3 for the input delay

. In absolute terms, an NRMSE of 0.3 is within the range of typically quoted best values (NRMSE = 0.15–0.4) [ 4 , 10 , 41 , 42 , 43 , 44 ], however, it is usually achieved with a much higher output dimension than the

used here. The performance achieved in this study came at a very low computational cost.…”

Section: Resultssupporting

confidence: 68%

Reservoir Computing with Delayed Input for Fast and Easy Optimisation

Jaurigue

Robertson

Wolters

et al. 2021

Entropy

Self Cite

View full text Add to dashboard Cite

Reservoir computing is a machine learning method that solves tasks using the response of a dynamical system to a certain input. As the training scheme only involves optimising the weights of the responses of the dynamical system, this method is particularly suited for hardware implementation. Furthermore, the inherent memory of dynamical systems which are suitable for use as reservoirs mean that this method has the potential to perform well on time series prediction tasks, as well as other tasks with time dependence. However, reservoir computing still requires extensive task-dependent parameter optimisation in order to achieve good performance. We demonstrate that by including a time-delayed version of the input for various time series prediction tasks, good performance can be achieved with an unoptimised reservoir. Furthermore, we show that by including the appropriate time-delayed input, one unaltered reservoir can perform well on six different time series prediction tasks at a very low computational expense. Our approach is of particular relevance to hardware implemented reservoirs, as one does not necessarily have access to pertinent optimisation parameters in physical systems but the inclusion of an additional input is generally possible.

show abstract

“…The inclusion of delayed input significantly reduced the NARMA10 error, reaching an NRMSE of about 0.3 for the input delay

used here. The performance achieved in this study came at a very low computational cost.…”

Section: Resultssupporting

confidence: 68%

Reservoir Computing with Delayed Input for Fast and Easy Optimisation

Jaurigue

Robertson

Wolters

et al. 2021

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…In tandem, physical implementations of reservoir computing in photonic [42][43][44] , memristive 45 , and neuromorphic 46 systems provide low-power alternatives to traditional computing hardware. Each application is accompanied by its own unique set of theoretical considerations and limitations 47 , thereby emphasizing the need for the underlying analytical mechanisms to make meaningful generalizations across such a wide range of systems.…”

Section: Discussionmentioning

confidence: 99%

Learning Continuous Chaotic Attractors with a Reservoir Computer

Smith,

Kim,

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural systems are well known for their ability to learn and store information as memories. Even more impressive is their ability to abstract these memories to create complex internal representations, enabling advanced functions such as the spatial manipulation of mental representations. While recurrent neural networks (RNNs) are capable of representing complex information, the exact mechanisms of how dynamical neural systems perform abstraction are still not well-understood, thereby hindering the development of more advanced functions. Here, we train a 1000-neuron RNN -a reservoir computer (RC) -to abstract a continuous dynamical attractor memory from isolated examples of dynamical attractor memories. Further, we explain the abstraction mechanism with new theory. By training the RC on isolated and shifted examples of either stable limit cycles or chaotic Lorenz attractors, the RC learns a continuum of attractors, as quantified by an extra Lyapunov exponent equal to zero. We propose a theoretical mechanism of this abstraction by combining ideas from differentiable generalized synchronization and feedback dynamics. Our results quantify abstraction in simple neural systems, enabling us to design artificial RNNs for abstraction, and leading us towards a neural basis of abstraction.Neural systems learn and store information as memories, and can even create abstract representations from these memories, such as how the human brain can change the pitch of a song or predict different trajectories of a moving object. Because we do not know how neurons work together to generate abstractions, we are unable to optimize artificial neural networks for abstraction, or directly measure abstraction in biological neural networks. Our goal is to provide a theory for how a simple neural network learns to abstract information from its inputs. We demonstrate that abstraction is possible using a simple neural network, and that abstraction can be quantified and measured using existing tools. Further, we provide a new mathematical mechanism for abstraction in artificial neural networks, allowing for future research applications in neuroscience and machine learning.

show abstract

“…Machine Learning is another rapidly developing application area of delay systems [29][30][31][32][33][34][35][36][37][38][39][40][41][42][43][44][45][46][47]. It is shown recently that DDEs can successfully realize a reservoir computing setup, theoretically [41][42][43][44]46,[48][49][50], and implemented in optoelectronic hardware [30,32,39]. In time-delay reservoir computing, a single DDE with either one or a few variables is used for building a ring network of coupled maps with fixed internal weights and fixed input weights.…”

Section: Introductionmentioning

confidence: 99%

Emulating complex networks with a single delay differential equation

Stelzer

Yanchuk

2021

Eur. Phys. J. Spec. Top.

View full text Add to dashboard Cite

A single dynamical system with time-delayed feedback can emulate networks. This property of delay systems made them extremely useful tools for Machine-Learning applications. Here, we describe several possible setups, which allow emulating multilayer (deep) feed-forward networks as well as recurrent networks of coupled discrete maps with arbitrary adjacency matrix by a single system with delayed feedback. While the network’s size can be arbitrary, the generating delay system can have a low number of variables, including a scalar case.

show abstract

Limitations of the Recall Capabilities in Delay-Based Reservoir Computing Systems

Cited by 27 publications

References 38 publications

Reservoir Computing with Delayed Input for Fast and Easy Optimisation

Reservoir Computing with Delayed Input for Fast and Easy Optimisation

Learning Continuous Chaotic Attractors with a Reservoir Computer

Emulating complex networks with a single delay differential equation

Contact Info

Product

Resources

About