Pattern recognition in Deep Boltzmann machines

Abstract

We consider a multi-layer Sherrington-Kirkpatrick spin-glass as a model for deep restricted Boltzmann machines and we solve for its quenched free energy, in the thermodynamic limit and allowing for a first step of replica symmetry breaking. This result is accomplished rigorously exploiting interpolating techniques and recovering the expression already known for the replica-symmetry case. Further, we drop the restriction constraint by introducing intra-layer connections among spins and we show that the resulting system can be mapped into a modular Hopfield network, which is also addressed rigorously via interpolating techniques up to the first step of replica symmetry breaking.

Free energy in multi-species mixed $p$-spin spherical models

“…Since (1.7) tells us that the Gaussian field H N is governed by these overlaps, it can be intuited that the free energy F N is related to the law of the array R = (R ℓ,ℓ ′ ) ℓ,ℓ ′ ≥1 , which we denote by Law(R; G N ). 1 The Parisi formula (1.16) makes the relationship between this law and lim N →∞ EF N precise, and this will be enough since it is a standard fact that F N concentrates around its mean (see Lemma A.2). But understanding this relationship-and indeed proving itrequires that we develop two fundamental concepts, namely (i) how the overlap distribution Law(R; G N ) is identified with some pair (ζ, Φ); and (ii) how the Parisi functional P emerges as the correct objective function.…”

“…In the Ising case, there are conjectured formulas for the limiting free energy [18,17,51] of the bipartite SK model, although not much is known rigorously. See [4,7,35,1] for results on a generalization of the bipartite SK model, and [5,6] for its restriction to a special subset of phase space.…”

“…In the next section we will select the u p,q parameters randomly, in which case E u will denote expectation with respect to the product measure P u under which each u p,q is a uniform random variable on [1,2], independent of all other variables. We will continue to write E for expectation over all Gaussian processes (and Poisson-Dirichlet cascades whenever they are present) with fixed u = (u p,q ) p,q≥1 .…”

“…As before, let us think of the perturbation parameters (u p,q ) p,q≥1 as i.i.d. uniform random variables on [1,2] which are independent of everything else, and then we write E u to denote expectation over all u p,q . With this modified viewpoint, we recall the definitions from (3.8) and apply (1.22) to the sequence a N = E u (E log ZN ), resulting in lim inf…”

“…. , a (1) m,n (κ)κ m−1 , κ m ), (5.17) Note that σ(s) does not in general belong to S N s (we only know σ(s) 2 2 ≤ ρ(s) 2 2 = N s + M s ), hence the decoration by a tilde. Therefore, we wish to perform the change of variables (5.16) for each species s ∈ S .…”