Orthogonal Echo State Networks and Stochastic Evaluations of Likelihoods

Mayer, Norbert Michael; Yu, Ying-Hao

doi:10.1007/s12559-017-9466-4

Cited by 4 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interesting applications of arithmetic encoders can be found in, e.g., [30], [31], [32], [33]. Recent investigations have revealed an analogy between arithmetic encoders and optimal ESNs [6], [34], [9], [35], [5], [36]. Making use of this idea, the connectivity in an RNN would be arranged such that for the given input statistics in U, the network performs a data compression so that as many as possible past input values are represented in the reservoir.…”

Section: Rnns and Their Fractal Dimensionmentioning

confidence: 99%

“…O NE of the key problems for Recurrent Neural Network (RNN) initialization, and in particular for Reservoir Computing (RC) methods like Echo State Networks (ESNs) [1], [2], [3] is that it is still unclear what kind of connectivity results in the best performance. For recurrent neural networks, some heuristics have shown much better performance than others: Dependent on the task, ESNs with orthogonal recurrent connectivity matrices that scale with the size of the network have shown better performance than other strategies [4], [5], [6]. An initialization method that can create a good recurrent weight matrix for a given task would be a compromise between an arbitrarily chosen connectivity matrix, and a connectivity matrix found by more expensive optimization, e.g., by backpropagation through time.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Analyzing Echo-state Networks Using Fractal Dimension

Mayer¹,

Obst²

2022

Preprint

View full text Add to dashboard Cite

This work joins aspects of reservoir optimization, information-theoretic optimal encoding, and at its center fractal analysis. We build on the observation that, due to the recursive nature of recurrent neural networks, input sequences appear as fractal patterns in their hidden state representation. These patterns have a fractal dimension that is lower than the number of units in the reservoir. We show potential usage of this fractal dimension with regard to optimization of recurrent neural network initialization. We connect the idea of "ideal" reservoirs to lossless optimal encoding using arithmetic encoders. Our investigation suggests that the fractal dimension of the mapping from input to hidden state shall be close to the number of units in the network. This connection between fractal dimension and network connectivity is an interesting new direction for recurrent neural network initialization and reservoir computing.

show abstract

Section: Rnns and Their Fractal Dimensionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Analyzing Echo-state Networks Using Fractal Dimension

Mayer¹,

Obst²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…entries drawn from uniform or Gaussian distributions [31,32]. However, in the literature it is possible to find reservoirs with different connection patterns, including deterministic topologies [50] and those exploiting the norm-preserving property of orthogonal matrices [36]. In our case the read-out matrix W o ∈ R No×Nr is optimised for the task at hand.…”

Section: Echo State Networkmentioning

confidence: 99%

Interpreting Recurrent Neural Networks Behaviour via Excitable Network Attractors

2019

View full text Add to dashboard Cite

Introduction: Machine learning provides fundamental tools both for scientific research and for the development of technologies with significant impact on society. It provides methods that facilitate the discovery of regularities in data and that give predictions without explicit knowledge of the rules governing a system. However, a price is paid for exploiting such flexibility: machine learning methods are typically black-boxes where it is difficult to fully understand what the machine is doing or how it is operating. This poses constraints on the applicability and explainability of such methods. Methods: Our research aims to open the black-box of recurrent neural networks, an important family of neural networks used for processing sequential data. We propose a novel methodology that provides a mechanistic interpretation of behaviour when solving a computational task. Our methodology uses mathematical constructs called excitable network attractors, which are invariant sets in phase space composed of stable attractors and excitable connections between them. Results and Discussion: As the behaviour of recurrent neural networks depends both on training and on inputs to the system, we introduce an algorithm to extract network attractors directly from the trajectory of a neural network while solving tasks. Simulations conducted on a controlled benchmark task confirm the relevance of these attractors for interpreting the behaviour of recurrent neural networks, at least for tasks that involve learning a finite number of stable states and transitions between them.as reservoir computing [32,33]. Echo state networks (ESNs) [21,30] constitute an important example of reservoir computing, where a recurrent layer (called a reservoir) is composed of a large number of neurons with randomly initialised connections that are not fine-tuned via gradient-based optimisation mechanisms. The main idea behind ESNs is to exploit the rich dynamics generated by the reservoir with an output layer, the read-out that is optimised to solve a specific task. Problem statement and research hypothesisThe high-dimensional and non-linear nature of RNNs complicates interpretability of their internal dynamics, which are characterised by complex, input-dependent spatio-temporal patterns of activity [47,55]. This poses constraints on understanding the behaviour of RNNs: they are usually viewed as black-boxes from which it is hard to extract useful knowledge about their inner workings. As highlighted by recent research efforts [10,24,40], similar interpretability issues affect many other machine learning methods. Furthermore, an increasing societal need to develop accountability and explainability of decision making by AI [17] is driving the development of methodologies for explaining the behaviour of such methods.Our aim in this paper is to develop effective models that capture the essential dynamical behaviour of RNNs on computational tasks as input-driven responses of a dynamical system, while neglecting microscopic details of the RNN dynamics in phase...

show abstract

Analyzing Echo-state Networks Using Fractal Dimension

Mayer

Obst

2022

2022 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Orthogonal Echo State Networks and Stochastic Evaluations of Likelihoods

Cited by 4 publications

References 21 publications

Analyzing Echo-state Networks Using Fractal Dimension

Analyzing Echo-state Networks Using Fractal Dimension

Interpreting Recurrent Neural Networks Behaviour via Excitable Network Attractors

Analyzing Echo-state Networks Using Fractal Dimension

Contact Info

Product

Resources

About