Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones

Abstract

The standard Hopfield model for associative neural networks accounts for biological Hebbian learning and acts as the harmonic oscillator for pattern recognition, however its maximal storage capacity is α ∼ 0.14, far from the theoretical bound for symmetric networks, i.e. α = 1. Inspired by sleeping and dreaming mechanisms in mammal brains, we propose an extension of this model displaying the standard on-line (awake) learning mechanism (that allows the storage of external information in terms of patterns) and an off-line (sleep) unlearning&consolidating mechanism (that allows spurious-pattern removal and pure-pattern reinforcement): this obtained daily prescription is able to saturate the theoretical bound α = 1, remaining also extremely robust against thermal noise. Both neural and synaptic features are analyzed both analytically and numerically. In particular, beyond obtaining a phase diagram for neural dynamics, we focus on synaptic plasticity and we give explicit prescriptions on the temporal evolution of the synaptic matrix. We analytically prove that our algorithm makes the Hebbian kernel converge with high probability to the projection matrix built over the pure stored patterns. Furthermore, we obtain a sharp and explicit estimate for the "sleep rate" in order to ensure such a convergence. Finally, we run extensive numerical simulations (mainly Monte Carlo sampling) to check the approximations underlying the analytical investigations (e.g., we developed the whole theory at the so called replica-symmetric level, as standard in the Amit-Gutfreund-Sompolinsky reference framework) and possible finite-size effects, finding overall full agreement with the theory.

Dreaming neural networks: rigorous results

“…the standard equivalence between Hopfield models and two-layers Boltzmann machines [16,18]), layer is equipped with imaginary real-valued neurons (best suitable to perform spectral analysis 10 ). As a consequence, the resulting interpolating architecture is rather tricky, by far richer than its classical limit yet it turns out to be managable and actually a sum rule for the quenched free energy related to the model can be written and even integrated, under the assumption of replica symmetry: such an expression, as well as those stemming from its extremization for the order parameters, sharply coincides with previous results [33], confirming them in each detail.…”

“…In the current work we mathematically described the phenomena of reinforcement and remotion, as pioneered by Crick & Mitchinson [29], by Hopfield [40] and by many others in the neuroscience literature, see e.g [30,38,47,48]): interestingly, such mechanisms have been evidenced to lead to an improvement of the retrieval capacity of the system. In particular, in [33], we showed that the system reaches the expected upper critical capacity α c = 1, still preserving robustness with respect to fast noise. However, the statistical mechanical analysis, set at the standard replica symmetric level of description, was carried out via non-rigorous approaches (e.g., replica trick and numerical simulations).…”

“…Then, we generalize the stochastic stability technique, introduced in [8,28] to address Sherrington-Kirkpatrick spin-glass and later developed in [19] to account also for bipartite spin-glasses (namely restricted Boltzmann machines or Hopfield networks [16] in a machine learning jargon [36,59]), so that it can as well deal with the present tripartite and correlated spin-glass. Next, by using this novel approach -that is mathematically well controllable at any stage of the calculations-we obtain the expression of the quenched replica-symmetric free energy related to the model (as well as the set of self-consistent equations for the order parameters) and we show that the resulting picture sharply coincides with that obtained via the replica-trick analysis [33]. This implies, in a cascade fashion, that all the results previously heuristically derived are actually proved (the most remarkable one being the saturation of the critical capacity).…”

“…Remark 2. Note that the matrix ξ T 1+t I+tC ξ, encoding the neuronal coupling, recovers the Hebbian kernel for t = 0 , while it approaches the pseudo-inverse matrix for t → ∞ (see [33] for the proof ). Accordingly, the model described by the Hamiltonian (2.1) spans, respectively, from the standard Hopfield model (t → 0) to the Kanter-Sompolinksy model [43] (t → ∞).…”

“…To match exactly the notation in [33] there is still a short way to go: it is convenient to re-scale m, p and X as 22) as this allows us to introduce the composite order parameter ∆ = 1 − αβ 2 b 2 (X − p) used in [33]. After these transformations, remembering the definition of the free energy (see (2.5)) and the definition of (a, b) (see (2.8)), we obtain exactly the same expression for the quenched free energy as that achieved in [33] via the replica trick, as stated by the next main Theorem 1. In the infinite volume limit, the replica symmetric free energy related to the neural network defined by eq.…”