Towards Real-World Applications of Online Learning Spiral Recurrent Neural Networks

Sollacher, Rudolf; Gao, Hongming

doi:10.4236/jilsa.2009.11001

Cited by 3 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors in [16] claims that the maximum P scales as N/ log(N). More recently, the authors in [17] discussed how to design networks of specific topologies, and how to reach α c values larger than 0.14. Unfortunately, it still finds a bound on the number of limiting behaviours that scale linearly to N. Clearly, the optimal storage problem is still open and how to achieve this optimal storage is still the subject of research [18].…”

Section: Introductionmentioning

confidence: 99%

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Gosti

Folli

Leonetti

et al. 2019

Entropy

View full text Add to dashboard Cite

In a neural network, an autapse is a particular kind of synapse that links a neuron onto itself. Autapses are almost always not allowed neither in artificial nor in biological neural networks. Moreover, redundant or similar stored states tend to interact destructively. This paper shows how autapses together with stable state redundancy can improve the storage capacity of a recurrent neural network. Recent research shows how, in an N-node Hopfield neural network with autapses, the number of stored patterns (P) is not limited to the well known bound 0.14 N , as it is for networks without autapses. More precisely, it describes how, as the number of stored patterns increases well over the 0.14 N threshold, for P much greater than N, the retrieval error asymptotically approaches a value below the unit. Consequently, the reduction of retrieval errors allows a number of stored memories, which largely exceeds what was previously considered possible. Unfortunately, soon after, new results showed that, in the thermodynamic limit, given a network with autapses in this high-storage regime, the basin of attraction of the stored memories shrinks to a single state. This means that, for each stable state associated with a stored memory, even a single bit error in the initial pattern would lead the system to a stationary state associated with a different memory state. This thus limits the potential use of this kind of Hopfield network as an associative memory. This paper presents a strategy to overcome this limitation by improving the error correcting characteristics of the Hopfield neural network. The proposed strategy allows us to form what we call an absorbing-neighborhood of state surrounding each stored memory. An absorbing-neighborhood is a set defined by a Hamming distance surrounding a network state, which is an absorbing because, in the long-time limit, states inside it are absorbed by stable states in the set. We show that this strategy allows the network to store an exponential number of memory patterns, each surrounded with an absorbing-neighborhood with an exponentially growing size.

show abstract

Section: Introductionmentioning

confidence: 99%

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Gosti

Folli

Leonetti

et al. 2019

Entropy

View full text Add to dashboard Cite

show abstract

“…Soon after, Mc Eliece et al (1987), considering only the Hebbian dyadic form for the coupling matrix, found a more severe limitation: the maximum P scales as N /log( N ). In a more recent study, Sollacher et al (2009) designed a network of specific topology, reaching α c -values larger than 0.14, but still maintaining the limit of a linear N dependence of the maximum storage capacity. The storage problem remains an open research question (Brunel, 2016).…”

Section: Introductionmentioning

confidence: 99%

On the Maximum Storage Capacity of the Hopfield Model

Folli

Leonetti

Ruocco

2017

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Recurrent neural networks (RNN) have traditionally been of great interest for their capacity to store memories. In past years, several works have been devoted to determine the maximum storage capacity of RNN, especially for the case of the Hopfield network, the most popular kind of RNN. Analyzing the thermodynamic limit of the statistical properties of the Hamiltonian corresponding to the Hopfield neural network, it has been shown in the literature that the retrieval errors diverge when the number of stored memory patterns (P) exceeds a fraction (≈ 14%) of the network size N. In this paper, we study the storage performance of a generalized Hopfield model, where the diagonal elements of the connection matrix are allowed to be different from zero. We investigate this model at finite N. We give an analytical expression for the number of retrieval errors and show that, by increasing the number of stored patterns over a certain threshold, the errors start to decrease and reach values below unit for P ≫ N. We demonstrate that the strongest trade-off between efficiency and effectiveness relies on the number of patterns (P) that are stored in the network by appropriately fixing the connection weights. When P≫N and the diagonal elements of the adjacency matrix are not forced to be zero, the optimal storage capacity is obtained with a number of stored memories much larger than previously reported. This theory paves the way to the design of RNN with high storage capacity and able to retrieve the desired pattern without distortions.

show abstract

“…Neural networks are widely used in the characterization of nonlinear systems [1][2][3][4][5][6][7], time-varying time-delay nonlinear systems [8] and they are applied in various applications [9][10][11][12].…”

Section: Introductionmentioning

confidence: 99%

A Comparative Study of Nonlinear Time-Varying Process Modeling Techniques: Application to Chemical Reactor

Errachdi¹,

Saad²,

Benrejeb³

2012

JILSA

View full text Add to dashboard Cite

This paper proposes the design and a comparative study of two nonlinear systems modeling techniques. These two approaches are developed to address a class of nonlinear systems with time-varying parameter. The first is a Radial Basis Function (RBF) neural networks and the second is a Multi Layer Perceptron (MLP). The MLP model consists of an input layer, an output layer and usually one or more hidden layers. However, training MLP network based on back propagation learning is computationally expensive. In this paper, an RBF network is called. The parameters of the RBF model are optimized by two methods: the Gradient Descent (GD) method and Genetic Algorithms (GA). However, the MLP model is optimized by the Gradient Descent method. The performance of both models are evaluated first by using a numerical simulation and second by handling a chemical process known as the Continuous Stirred Tank Reactor CSTR. It has been shown that in both validation operations the results were successful. The optimized RBF model by Genetic Algorithms gave the best results.

show abstract

Towards Real-World Applications of Online Learning Spiral Recurrent Neural Networks

Abstract: ABSTRACT

Cited by 3 publications

References 13 publications

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

On the Maximum Storage Capacity of the Hopfield Model

A Comparative Study of Nonlinear Time-Varying Process Modeling Techniques: Application to Chemical Reactor

Contact Info

Product

Resources

About