The signal-to-noise analysis of the Little–Hopfield model revisited

Abstract

Using the generating functional analysis an exact recursion relation is derived for the time evolution of the effective local field of the fully connected Little-Hopfield model. It is shown that, by leaving out the feedback correlations arising from earlier times in this effective dynamics, one precisely finds the recursion relations usually employed in the signal-to-noise approach. The consequences of this approximation as well as the physics behind it are discussed. In particular, it is pointed out why it is hard to notice the effects, especially for model parameters corresponding to retrieval. Numerical simulations confirm these findings. The signal-to-noise analysis is then extended to include all correlations, making it a full theory for dynamics at the level of the generating functional analysis. The results are applied to the frequently employed extremely diluted (a)symmetric architectures and to sequence processing networks.

Transient dynamics of sparsely connected Hopfield neural networks with arbitrary degree distributions

“…Equilibrium properties of fully connected Hopfield neural networks have been well studied using spin-glass theory, especially the replica method [2,3]. Their dynamics is also studied using the generating functional method [4] and signal-to-noise analysis [5][6][7].Given the huge number of neurons, there is only a small number of interconnections in the human brain cortex (∼10 11 neurons and ∼10 14 synapses). In order to simulate a biological genuine model rather than the fully connected networks, various random diluted models were studied, including extremely diluted model [8,9], finite diluted model [10,11], and finite connection model [12,13].…”

“…(14)- (17), one can calculate temporal evolution of overlap parameters up to an arbitrary time step. Using auxiliary thermal fields γ (t) to express the stochastic dynamics [7], it is easy to extend the method to arbitrary temperatures by averaging the zero temperature results over the auxiliary fields.…”

Signal-to-noise analysis of Hopfield neural networks with a formulation of the dynamics in terms of transition probabilities

“…In this case the crosstalk noise has a Gaussian distribution and the statistical neurodynamics gives exact results. For models that present symmetry in the synapses, we found feedback correlations and it can be shown [21,22] that, when some relevant feedback correlations are ignored, the results of the SNA method are only an approximation to the exact equations of the GFA method. Finding the relevant feedback correlations, for example in the fully connected Little-Hopfield model, is not a trivial task and the GFA method has served as a guide in this search.…”

“…Our motivation in this section is to obtain the local field in the thermodynamic limit N → ∞ utilizing the SNA method [15,16,22,17]. From the local field we can determine the relevant order parameters.…”

“…In Ref. [22], Bolle et al apply the SNA method with a formulation of the dynamics in terms of gain functions in the fully connected Little-Hopfield model. A detailed and extensive comparative analysis shows that the SNA method can be extended, making it a full theory for dynamics of Hopfield neural networks with infinite connectivity.…”