High storage capacity in the Hopfield model with auto-interactions—stability analysis

Rocchi, Jacopo; Saad, David; Tantari, Daniele

doi:10.1088/1751-8121/aa8fd7

Cited by 9 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the contrary, when it allows autapses, which implies that the diagonal elements in the connectivity matrix J may be different from zero, it finds that a new region exists, with P N, where the number of retrieval errors decreases on increasing P. In this new region, the number of retrieval errors reaches values lower than one, thus it finds new favorable conditions for an effective and efficient storage of memory patterns in an RNN. Moreover, it shows that, in this region, the number of storable patterns grows exponentially with N. In response to these results, Rocchi et al [20] found that, in the thermodynamic limit at P N, the basin of attraction of the stationary states vanishes. Thus, asymptotically, the basing of attraction of the stationary state coincides with the stationary state itself.…”

Section: Introductionmentioning

confidence: 88%

See 1 more Smart Citation

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Gosti

Folli

Leonetti

et al. 2019

Entropy

View full text Add to dashboard Cite

In a neural network, an autapse is a particular kind of synapse that links a neuron onto itself. Autapses are almost always not allowed neither in artificial nor in biological neural networks. Moreover, redundant or similar stored states tend to interact destructively. This paper shows how autapses together with stable state redundancy can improve the storage capacity of a recurrent neural network. Recent research shows how, in an N-node Hopfield neural network with autapses, the number of stored patterns (P) is not limited to the well known bound 0.14 N , as it is for networks without autapses. More precisely, it describes how, as the number of stored patterns increases well over the 0.14 N threshold, for P much greater than N, the retrieval error asymptotically approaches a value below the unit. Consequently, the reduction of retrieval errors allows a number of stored memories, which largely exceeds what was previously considered possible. Unfortunately, soon after, new results showed that, in the thermodynamic limit, given a network with autapses in this high-storage regime, the basin of attraction of the stored memories shrinks to a single state. This means that, for each stable state associated with a stored memory, even a single bit error in the initial pattern would lead the system to a stationary state associated with a different memory state. This thus limits the potential use of this kind of Hopfield network as an associative memory. This paper presents a strategy to overcome this limitation by improving the error correcting characteristics of the Hopfield neural network. The proposed strategy allows us to form what we call an absorbing-neighborhood of state surrounding each stored memory. An absorbing-neighborhood is a set defined by a Hamming distance surrounding a network state, which is an absorbing because, in the long-time limit, states inside it are absorbed by stable states in the set. We show that this strategy allows the network to store an exponential number of memory patterns, each surrounded with an absorbing-neighborhood with an exponentially growing size.

show abstract

Section: Introductionmentioning

confidence: 88%

“…We thus bound the maximum number of storable points with Ω TE [13]. Unfortunately, as pointed out by Rocchi et al [20], the basin of attraction of these points shrinks to the point that it contains only the stable states. In the lower panel, we show how, with our approach, we overcome this limit.…”

Section: Storing Patternsmentioning

confidence: 93%

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Gosti

Folli

Leonetti

et al. 2019

Entropy

View full text Add to dashboard Cite

show abstract

“…For example, the pseudo-inverse rule (also called projection rule) can increase HNNs storage capacity and improve accuracy (Wu et al, 2012;Sahoo et al, 2016), but is not local nor incremental. Moreover, recent works have explored learning rules with self-feedback connections (non-0 diagonal), and have shown higher accuracy for a high number of stored patterns (Liou and Yuan, 1999;Folli et al, 2017;Rocchi et al, 2017;Gosti et al, 2019). In summary, despite the present limitations of the ONN, features in terms of FPS, computation time and training, are encouraging toward the exploration of a wider range of applications.…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

Digital Implementation of Oscillatory Neural Network for Image Recognition Applications

et al. 2021

View full text Add to dashboard Cite

Computing paradigm based on von Neuman architectures cannot keep up with the ever-increasing data growth (also called “data deluge gap”). This has resulted in investigating novel computing paradigms and design approaches at all levels from materials to system-level implementations and applications. An alternative computing approach based on artificial neural networks uses oscillators to compute or Oscillatory Neural Networks (ONNs). ONNs can perform computations efficiently and can be used to build a more extensive neuromorphic system. Here, we address a fundamental problem: can we efficiently perform artificial intelligence applications with ONNs? We present a digital ONN implementation to show a proof-of-concept of the ONN approach of “computing-in-phase” for pattern recognition applications. To the best of our knowledge, this is the first attempt to implement an FPGA-based fully-digital ONN. We report ONN accuracy, training, inference, memory capacity, operating frequency, hardware resources based on simulations and implementations of 5 × 3 and 10 × 6 ONNs. We present the digital ONN implementation on FPGA for pattern recognition applications such as performing digits recognition from a camera stream. We discuss practical challenges and future directions in implementing digital ONN.

show abstract

“…When a neural system is used as an associated memory, the stable equilibria of the neural network correspond to the static retrievable memory. So far, many scholars devoted their efforts to establish a neural network with coexisting multiple equilibria [12][13][14], which denotes the storage capacity of a neural system [15,16]. For the low-dimensional Hopfield neural network system, the existing dynamical analysis is focused on the local stability and Hopf bifurcation of the trivial equilibrium [17][18][19].…”

Section: Introductionmentioning

confidence: 99%

Multiple bifurcations and periodic coexistence in a delayed Hopfield two-neural system with a monotonic activation function

et al. 2019

View full text Add to dashboard Cite

In this paper, we consider a delayed Hopfield two-neural system with a monotonic activation function and find the periodic coexistence by bifurcation analysis. Firstly, we obtain the pitchfork bifurcation of the trivial equilibrium employing the central manifold and normal form methods. The neural system exhibits two pitchfork bifurcations near the trivial equilibrium. Then, analyzing the characteristic equation of the nontrivial equilibrium, we illustrate the saddle-node bifurcation of the nontrivial equilibria. The system exhibits the multi-coexistences of the stable and unstable equilibria. Further, we illustrate the plane regions of parameters having different numbers of equilibria. To obtain a time delay in neural system dynamics, we present the stability analysis and find the periodic orbit. The system exhibits stability switching by the Hopf bifurcation curves. Finally, the dynamic behaviors near the Hopf-Hopf bifurcation point are presented. The system exhibits coexistence of multiple periodic orbits with different frequencies.

show abstract

High storage capacity in the Hopfield model with auto-interactions—stability analysis

Cited by 9 publications

References 13 publications

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Digital Implementation of Oscillatory Neural Network for Image Recognition Applications

Multiple bifurcations and periodic coexistence in a delayed Hopfield two-neural system with a monotonic activation function

Contact Info

Product

Resources

About