Echo State Condition at the Critical Point

Mayer, Norbert Michael

doi:10.3390/e19010003

Cited by 8 publications

(7 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In any case the ESP is fulfilled. This can be achieved by using matrices that have a maximal singular value that is lower or equal to one [16]. We use here in all simulations recurrent matrices with a maximal singular value of one.…”

Section: Recurrent Connectivitymentioning

confidence: 99%

“…Theoretical reasoning behind that has been explained in the case of linear networks [25]. Technically normal matrices have the virtue [15,16] that all absolute eigenvalues are equal to one. Thus, transferring information by executing the linear part of the update rule (cf.…”

Section: Recurrent Connectivitymentioning

confidence: 99%

“…Connections in the output layer are trained to reproduce the training signal. It may be interpreted as a kind of interpolation that uses a recurrent random kernel [16]. The network dynamics are defined for discrete time-steps t, with the following equations:…”

Section: Esn Structurementioning

confidence: 99%

See 2 more Smart Citations

Orthogonal Echo State Networks and Stochastic Evaluations of Likelihoods

Mayer

2017

Cogn Comput

View full text Add to dashboard Cite

We report about probabilistic likelihood estimates that are performed on time series using an echo state network with orthogonal recurrent connectivity. The results from tests using synthetic stochastic input time series with temporal inference indicate that the capability of the network to infer depends on the balance between input strength and recurrent activity. This balance has an influence on the network with regard to the quality of inference from the short term input history versus inference that accounts for influences that date back a long time. Sensitivity of such networks against noise and the finite accuracy of network states in the recurrent layer are investigated. In addition, a measure based on mutual information between the output time series and the reservoir is introduced. Finally, different types of recurrent connectivity are evaluated. Orthogonal matrices show the best results of all investigated connectivity types overall, but also in the way how the network performance scales with the size of the recurrent layer. Structured abstractBackground. There is still a large gap between reservoir computing, probability theory and measures of information processing. The purpose of this paper is to provide an as simple as possible mean to calculate probabilistic likelihoods for events with regard to a target function given the input history.Methods. The results of the article have been achieved by symbolic and numerical analysis, as outlined in the paper.Results. The paper analytically argues for a simple way to calculate probabilistic likelihoods for a training output. As a first application this technique has been used to compute a lower limit estimate for the mutual information between a synthetic time series and a reservoir with orthogonal recurrent connectivity and other types of recurrent connectivity. The results indicate that the optimal performance depends on the way of balancing the input strength with the recurrent activity, which also has an influence on the network with regard to the quality of the inference from short term input history versus inference that accounts for influences that date back a long time in the input history. Finally, sensitivity of such networks against noise and the finite accuracy of network states in the recurrent layer are investigated.Conclusions. Methods in this paper describe and investigate in detail a potential link between reservoir computing and probabilistic modeling. The results argue for the virtue of using orthogonal matrices for recurrent connectivity. Those reservoirs show a very satisfying performance that also scales very nicely with the number of the neurons in the recurrent layer, which is not necessarily the case for general recurrent connectivity.

show abstract

Section: Recurrent Connectivitymentioning

confidence: 99%

Section: Recurrent Connectivitymentioning

confidence: 99%

See 1 more Smart Citation

Orthogonal Echo State Networks and Stochastic Evaluations of Likelihoods

Mayer

2017

Cogn Comput

View full text Add to dashboard Cite

show abstract

“…In an ecological context, the inertia is related to an impairment and competition between species. The study by Mayer [11] illustrates the effects of critical connectivity in echo state networks and identifies under which conditions the recurrent connectivity is achieved. The results are contrasted with alternative approaches considering the dynamics near the "edge of chaos".…”

mentioning

confidence: 99%

“…The next four papers [8][9][10][11] are placed in the complexity-criticality-computation overlap which is central to our issue. The study of Erten et al [8] continues the information-theoretic theme by applying the information dynamics framework to studies of critical thresholds during epidemics.…”

mentioning

confidence: 99%

Complexity, Criticality and Computation

Prokopenko

2017

Entropy

View full text Add to dashboard Cite

What makes a system "complex"? Is it merely the number of components it integrates, a nonlinear nature of the dependencies and feedbacks among its parts, or an unpredictable behavior it exhibits over time? The term "complexity" was initially applied generically to express the lack of predictability, reflecting on the self-organization of a synergistic macroscopic behavior out of interactions between the constituent microscopic parts, and the emergence of global patterns. Without a doubt, by now the concept has acquired a fairly definitive meaning, describing a distinct field of research and education and a new approach to science and engineering. There are abundant examples showing that the enterprise of Complex Systems, having achieved a substantial level of maturity, reaches back into our everyday lives, revealing patterns of complexity that should be considered without employing a reductionist logic [1].Similarly, the idea of criticality was originally motivated by studies of various crises and disruptive events, as well as sensitivities to initial conditions, but over time has developed into a precise field: critical dynamics. Research into critical dynamics is typically focused on the behavior of dynamical spatiotemporal systems during phase transitions where scale invariance prevails and symmetries break. Crucially, such behavior can be understood in terms of the control and order parameters. For instance, a second-order phase transition in a ferromagnetic system, separating two qualitatively different behaviors, can be reached by controlling the temperature parameter: the "disordered" and isotropic (symmetric) high-temperature phase is characterized by the absence of net magnetization, while the "ordered" and anisotropic (less symmetric) low-temperature phase can be described by an order parameter, the net magnetization vector defining a preferred direction in space. Critical phenomena have become associated with the physics of critical points, such as fractal behavior, the divergence of the correlation length, power-law divergences (e.g., the divergence of the magnetic susceptibility in the ferromagnetic phase transition), universality of relevant critical exponents, and so on. Now, these precise theoretical notions begin to reconnect with their motivating applied studies of crisis modeling, forecasting, and response. There is a growing awareness that complexity is strongly related to criticality, and many examples of self-organizing complex systems can be found in applications managing complexity specifically at critical regimes.A similar loop originating in practical studies, maturing to an exact science with precise but narrow definitions, and then reaching back to applied scenarios, can be seen in the realm of distributed computation. These days, complex systems can be viewed as distributed information-processing systems, in the domains ranging from systems biology and artificial life to computational neuroscience, digital circuitry, and transport networks [2]. Consciousness emerging from neuronal ac...

show abstract