Information storage in neural networks with low levels of activity

Abstract

The Hopfield model of a neural network is extended to allow for the storage and retrieval of biased patterns, [g'";), where N 'g, g =a is arbitrary. Such patterns represent levels of activity (i.e. , percentage of firing neurons) equal to 2 (1+a),-1 &a & 1. If the coupling constants (synaptic efficacies) are constructed as in the original Hopfield model, the system can retrieve at most a very small number of patterns (p & 1+a). This is due to the finite correlations (overlaps) between the patterns. The model is modified by subtracting the bias a from each pattern as it enters into the couplings. This modification restores the ability of the model to store a macroscopic number of patterns. Yet spurious states are found to plague the dynamics. It is then argued that the dynamics of the network should be consistent with the levels of activity of the stored patterns. This is implemented by adding a global constraint, which restricts the configuration space to states whose mean activity is in the neighborhood of~(1+a). The consequences of the restricted dynamics are analyzed in the replica symmetric mean-field theory. The global constraint suppresses spurious states and leads to the unexpected result that the storage capacity is higher than that of the unbiased network, up to very high values of the bias (~a~=0.99). However, the information content of such networks is shown to be a monotonically decreasing function of a.

Slow synaptic dynamics in a network: From exponential to power-law forgetting

“…We have so far described the various dynamical regimes characterising the evolution of the mean synaptic strength J(t) according to the mean-field dynamical equation (13). Here, we describe the regions of parameter space where these will be found.…”

“…In this section we illustrate the behaviour of the attractive fixed points of the mean-field dynamical equation (13), and of the corresponding relaxation times. The main emphasis is on the divergent behaviour of the relaxation times when the critical manifold or the tricritical point is approached.…”

“…The mean synaptic strength J(t) of the whole network will be unaffected by such a localised perturbation, and thus continue to obey (13). Let us again assume that the system has reached one of the fixedpoint values J.…”

“…The single collective degree of freedom of the model, namely its mean synaptic strength J(t), evolves according to equation (13), where the rate function…”

“…For example, the study of neural networks [6][7][8], while it greatly simplifies biological structures in order to make them tractable, has still been able to make an impact on the parent field. In particular, neural networks such as the Hopfield model [9,10] have been extensively investigated via methods borrowed from the statistical physics of disordered and complex systems [11][12][13][14]. In these models, memories are stored as patterns of neural activities, which correspond both to low-energy states and to attractors of the stochastic dynamics of the model.…”