Short term memory in input-driven linear dynamical systems

Tiňo, Peter; Rodan, Ali

doi:10.1016/j.neucom.2012.12.041

Cited by 18 publications

(12 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A comparison of how the measures of the information dynamics framework (Lizier et al, 2007(Lizier et al, , 2012, the information processing capacity for dynamical systems (Dambre et al, 2012), measures of criticality (Bertschinger and Natschläger, 2004;Prokopenko et al, 2011) or of memory capacity (Jaeger, 2001;Ganguli et al, 2008) relate to each other should reveal some interesting insights (see, e.g., Tino and Rodan, 2013), since they all cover some aspects of dynamical systems. Establishing the relation between the information dynamics framework, with recent extension for input-driven systems, and information processing capacity, for example, could help to overcome requirements for i.i.d.…”

Section: Resultsmentioning

confidence: 99%

Guided Self-Organization of Input-Driven Recurrent Neural Networks

Obst

Boedecker

2014

Guided Self-Organization: Inception

View full text Add to dashboard Cite

We review attempts that have been made towards understanding the computational properties and mechanisms of input-driven dynamical systems like RNNs, and reservoir computing networks in particular. We provide details on methods that have been developed to give quantitative answers to the questions above. Following this, we show how self-organization may be used to improve reservoirs for better performance, in some cases guided by the measures presented before. We also present a possible way to quantify task performance using an information-theoretic approach, and finally discuss promising future directions aimed at a better understanding of how these systems perform their computations and how to best guide self-organized processes for their optimization.Comment: 23 pages, to appear in M. Prokopenko (ed.), Guided Self-Organization: Inception, Springer, 201

show abstract

Section: Resultsmentioning

confidence: 99%

Guided Self-Organization of Input-Driven Recurrent Neural Networks

Obst

Boedecker

2014

Guided Self-Organization: Inception

View full text Add to dashboard Cite

show abstract

“…In order to guarantee asymptotic stability, ESNs must satisfy the so-called echo state property [44]- [47], which requires the reservoir exhibiting short-term memory (exponential fading) [48], [49]. Recently, in [50] the author investigated the effects of criticality in ESN memory, showing that, under certain conditions, the echo state property can still be verified even if the memory vanishes more slowly (i.e., following a power-law function).…”

Section: Echo State Networkmentioning

confidence: 99%

Determination of the Edge of Criticality in Echo State Networks Through Fisher Information Maximization

Livi

Bianchi

Alippi

2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Abstract-It is a widely accepted fact that the computational capability of recurrent neural networks (RNNs) is maximized on the so-called "edge of criticality." Once the network operates in this configuration, it performs efficiently on a specific application both in terms of: 1) low prediction error and 2) high shortterm memory capacity. Since the behavior of recurrent networks is strongly influenced by the particular input signal driving the dynamics, a universal, application-independent method for determining the edge of criticality is still missing. In this paper, we aim at addressing this issue by proposing a theoretically motivated, unsupervised method based on Fisher information for determining the edge of criticality in RNNs. It is proved that Fisher information is maximized for (finite-size) systems operating in such critical regions. However, Fisher information is notoriously difficult to compute and requires the analytic form of the probability density function ruling the system behavior. This paper takes advantage of a recently developed nonparametric estimator of the Fisher information matrix and provides a method to determine the critical region of echo state networks (ESNs), a particular class of recurrent networks. The considered control parameters, which indirectly affect the ESN performance, are explored to identify those configurations lying on the edge of criticality and, as such, maximizing Fisher information and computational performance. Experimental results on benchmarks and real-world data demonstrate the effectiveness of the proposed method.Index Terms-Echo state network (ESN), edge of criticality, Fisher information, nonparametric estimation.

show abstract

“…To specifically characterize capability of input-driven dynamical systems to keep in their state-space information about past inputs, several memory quantifiers were proposed, for example short term memory capacity [9] and Fisher memory curve [6]. Even though those two measures have been developed from completely different perspectives, deep connections exist between them [20]. The concept of memory capacity, originally developed for univariate input streams, was generalized to multivariate inputs in [8].…”

Section: Accepted Manuscriptmentioning

confidence: 99%

“…the expression of input weights v in this basis, i.e.ṽ = U T N v. It has been shown in [20] that for symmetric dynamic couplings,…”

Section: With a Zero-mean Distribution Of Finite Moments And Variancementioning

confidence: 99%

Asymptotic Fisher memory of randomized linear symmetric Echo State Networks

Tiňo

2018

Neurocomputing

Self Cite

View full text Add to dashboard Cite

Link to publication on Research at Birmingham portal General rights Unless a licence is specified above, all rights (including copyright and moral rights) in this document are retained by the authors and/or the copyright holders. The express permission of the copyright holder must be obtained for any use of this material other than for purposes permitted by law. • Users may freely distribute the URL that is used to identify this publication. • Users may download and/or print one copy of the publication from the University of Birmingham research portal for the purpose of private study or non-commercial research. • User may use extracts from the document in line with the concept of 'fair dealing' under the Copyright, Designs and Patents Act 1988 (?) • Users may not further distribute the material nor use it for the purposes of commercial gain. Where a licence is displayed above, please note the terms and conditions of the licence govern your use of this document. When citing, please reference the published version. Take down policy While the University of Birmingham exercises care and attention in making items available there are rare occasions when an item has been uploaded in error or has been deemed to be commercially or otherwise sensitive.

show abstract

Short term memory in input-driven linear dynamical systems

Cited by 18 publications

References 7 publications

Guided Self-Organization of Input-Driven Recurrent Neural Networks

Guided Self-Organization of Input-Driven Recurrent Neural Networks

Determination of the Edge of Criticality in Echo State Networks Through Fisher Information Maximization

Asymptotic Fisher memory of randomized linear symmetric Echo State Networks

Contact Info

Product

Resources

About