On the Maximum Storage Capacity of the Hopfield Model

Folli, Viola; Leonetti, Marco; Ruocco, G.

doi:10.3389/fncom.2016.00144

Cited by 36 publications

(37 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors in [19] give an analytical expression for the number of retrieval errors, and, when it restricts itself to the region P < N, it retrieves the bound P < α c N, which is analytically predicted in [11,12]. On the contrary, when it allows autapses, which implies that the diagonal elements in the connectivity matrix J may be different from zero, it finds that a new region exists, with P N, where the number of retrieval errors decreases on increasing P. In this new region, the number of retrieval errors reaches values lower than one, thus it finds new favorable conditions for an effective and efficient storage of memory patterns in an RNN.…”

Section: Introductionmentioning

confidence: 99%

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Gosti

Folli

Leonetti

et al. 2019

Entropy

Self Cite

View full text Add to dashboard Cite

In a neural network, an autapse is a particular kind of synapse that links a neuron onto itself. Autapses are almost always not allowed neither in artificial nor in biological neural networks. Moreover, redundant or similar stored states tend to interact destructively. This paper shows how autapses together with stable state redundancy can improve the storage capacity of a recurrent neural network. Recent research shows how, in an N-node Hopfield neural network with autapses, the number of stored patterns (P) is not limited to the well known bound 0.14 N , as it is for networks without autapses. More precisely, it describes how, as the number of stored patterns increases well over the 0.14 N threshold, for P much greater than N, the retrieval error asymptotically approaches a value below the unit. Consequently, the reduction of retrieval errors allows a number of stored memories, which largely exceeds what was previously considered possible. Unfortunately, soon after, new results showed that, in the thermodynamic limit, given a network with autapses in this high-storage regime, the basin of attraction of the stored memories shrinks to a single state. This means that, for each stable state associated with a stored memory, even a single bit error in the initial pattern would lead the system to a stationary state associated with a different memory state. This thus limits the potential use of this kind of Hopfield network as an associative memory. This paper presents a strategy to overcome this limitation by improving the error correcting characteristics of the Hopfield neural network. The proposed strategy allows us to form what we call an absorbing-neighborhood of state surrounding each stored memory. An absorbing-neighborhood is a set defined by a Hamming distance surrounding a network state, which is an absorbing because, in the long-time limit, states inside it are absorbed by stable states in the set. We show that this strategy allows the network to store an exponential number of memory patterns, each surrounded with an absorbing-neighborhood with an exponentially growing size.

show abstract

Section: Introductionmentioning

confidence: 99%

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Gosti

Folli

Leonetti

et al. 2019

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…Given any initial neural state, or input, a discrete-time recurrent neural network dynamically falls into an attractor. In this framework, the attractor is the retrieved limit behavior (Hebb, 1949;Amit et al, 1985b;McEliece et al, 1987;Folli et al, 2017;Gutfreund et al, 1988;Bastolla & Parisi, 1998;Sompolinsky et al, 1988;Wainrib & Touboul, 2013). Finally, it is important to consider that a recurrent network associates a limit behavior to each input from the set of all possible N -bit inputs, since the number of limit behaviors C is such that C << 2 N , it performs a many-to-few mapping.…”

Section: Introductionmentioning

confidence: 99%

Effect of dilution in asymmetric recurrent neural networks

et al. 2018

Self Cite

View full text Add to dashboard Cite

We study with numerical simulation the possible limit behaviors of synchronous discrete-time deterministic recurrent neural networks composed of N binary neurons as a function of a network's level of dilution and asymmetry. The network dilution measures the fraction of neuron couples that are connected, and the network asymmetry measures to what extent the underlying connectivity matrix is asymmetric. For each given neural network, we study the dynamical evolution of all the different initial conditions, thus characterizing the full dynamical landscape without imposing any learning rule. Because of the deterministic dynamics, each trajectory converges to an attractor, that can be either a fixed point or a limit cycle. These attractors form the set of all the possible limit behaviors of the neural network. For each network we then determine the convergence times, the limit cycles' length, the number of attractors, and the sizes of the attractors' basin. We show that there are two network structures that maximize the number of possible limit behaviors. The first optimal network structure is fully-connected and symmetric. On the contrary, the second optimal network structure is highly sparse and asymmetric. The latter optimal is similar to what observed in different biological neuronal circuits. These observations lead us to hypothesize that independently from any given learning model, an efficient and effective biologic network that stores a number of limit behaviors close to its maximum capacity tends to develop a connectivity structure similar to one of the optimal networks we found.

show abstract

“…Diagonal interaction terms were not considered in the early works about Hopfield model for a physical reason: in the corresponding spin models the field of a variable is induced by the state of its neighbours, but not on its own; thus self interactions do not exist and J ii = 0, ∀i. Neural networks with diagonal terms have been studied in [9] and a very interesting regime has been found for α 1. The probability p V that a given pattern ξ µ is a not fixed point of the dynamics has been computed and has been shown to be very small for very low α = P/N , as expected, but surprisingly another region has been identified at the very large α regime, where p V is also very small.…”

Section: Dynamics Of a Neural Networkmentioning

confidence: 99%

“…Even if this result seems to invalidate the usefulness of this new regime, it was shown that the ratiō p V /p V tends to a finite number, e, in the large N limit and that real patterns have an higher probability of being fixed points of the model. In the next section we address the stability question of these patterns, making use of the same strategy used in [9].…”

Section: Dynamics Of a Neural Networkmentioning

confidence: 99%

See 1 more Smart Citation

High storage capacity in the Hopfield model with auto-interactions—stability analysis

Rocchi

Saad

Tantari

2017

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

Recent studies point to the potential storage of a large number of patterns in the celebrated Hopfield associative memory model, well beyond the limits obtained previously. We investigate the properties of new fixed points to discover that they exhibit instabilities for small perturbations and are therefore of limited value as associative memories. Moreover, a large deviations approach also shows that errors introduced to the original patterns induce additional errors and increased corruption with respect to the stored patterns.

show abstract

On the Maximum Storage Capacity of the Hopfield Model

Cited by 36 publications

References 20 publications

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Effect of dilution in asymmetric recurrent neural networks

High storage capacity in the Hopfield model with auto-interactions—stability analysis

Contact Info

Product

Resources

About