Dreaming neural networks: rigorous results

Abstract

Recently a daily routine for associative neural networks has been proposed: the network Hebbian-learns during the awake state (thus behaving as a standard Hopfield model), then, during its sleep state, optimizing information storage, it consolidates pure patterns and removes spurious ones: this forces the synaptic matrix to collapse to the projector one (ultimately approaching the Kanter-Sompolinksy model). This procedure keeps the learning Hebbian-based (a biological must) but, by taking advantage of a (properly stylized) sleep phase, still reaches the maximal critical capacity (for symmetric interactions). So far this emerging picture (as well as the bulk of papers on unlearning techniques) was supported solely by mathematically-challenging routes, e.g. mainly replica-trick analysis and numerical simulations: here we rely extensively on Guerra's interpolation techniques developed for neural networks and, in particular, we extend the generalized stochastic stability approach to the case. Confining our description within the replica symmetric approximation (where the previous ones lie), the picture painted regarding this generalization (and the previously existing variations on theme) is here entirely confirmed. Further, still relying on Guerra's schemes, we develop a systematic fluctuation analysis to check where ergodicity is broken (an analysis entirely absent in previous investigations). Remarkably we find that, as long as the network is awake, ergodicity is bounded by the Amit-Gutfreund-Sompolinsky critical line (as it should), but, as the network sleeps, sleeping destroys spin glass states by extending both the retrieval as well as the ergodic region: after an entire sleeping session the solely surviving regions are retrieval and ergodic ones and this allows the network to achieve the perfect retrieval regime (where the number of storable patterns exactly equals the number of neurons the network is built of).

PDE/Statistical Mechanics Duality: Relation Between Guerra’s Interpolated p-Spin Ferromagnets and the Burgers Hierarchy

“…Indeed, as firstly suggested in [84], Statistical Mechanics turned out to be an effective tool in the description of universal phenomena in biological systems, see also [7,12,16,29,57,63,72,78]. For instance, kinetics of biochemical reactions can be framed in a purely statistical mechanics picture in terms of the Curie-Weiss ferromagnetic model, see [8,11]. However, since in standard statistical mechanics one considers the thermodynamic limit N → ∞, this scenario is a good approximation only for systems with large size.…”

Generalized Guerra’s interpolation schemes for dense associative neural networks

“…1, we reported the phase diagrams of the relativistic Hopeld model for various values of λ(=1, 2 and 5). As a rst comment, we see that increasing the parameter λ, the critical line for the transition to the ergodic region changes convexity, and the ergodic region gets wider towards smaller values of the noise (this features is in common with [47]). While the critical storage capacity remains α c (λ, β → ∞) ∼ 0.14, we see that the corresponding critical line (i.e.…”

“…We will not report the proof of this proposition as passages are quite lengthy but rather standard (see [19,47]), rather, to understand the physics underlying this streaming, it is enough to show how the core of these rules can be obtained in general. The simplest (already s-replicated) interpolating structure reads as…”

Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

“…In recent works ( 28 ) is slightly revised as [ 55 , 56 ] which, beyond the REM sleep, also accounts for the slow wave (SW) sleep; the former produces the removal of unnecessary memories, the latter the consolidation of important ones. The resulting model, referred to as reinforcement and removal (RR) model, extends the unlearning approaches by simultaneously doing remotion of spurious states and reinforcement of pure ones, providing extra stability of these states, finally resulting in a sensibly enlarged and more robust retrieval region.…”

“…In this way, we obtained a Boltzmann machine where we can set the number of hidden variable up to . This allows having many degrees of freedom, i.e., many coefficients that allow to represent reality without falling into the overfitting regime [ 55 ].…”