Capacity lower bound for the Ising perceptron

Abstract

We consider the Ising perceptron with gaussian disorder, which is equivalent to the discrete cube t´1,`1u N intersected by M random half-spaces. e perceptron's capacity is α N " M N {N for the largest integer M N such that the intersection in nonempty. It is conjectured by Krauth and Mézard (1989) that the (random) ratio α N converges in probability to an explicit constant α‹ . " 0.83. Kim and Roche (1998) proved the existence of a positive constant γ such that γ ď α N ď 1´γ with high probability; see also Talagrand (1999). In this paper we show that the Krauth-Mézard conjecture α‹ is a lower bound with positive probability, under the condition that an explicit univariate function S‹pλq is maximized at λ " 0. Our proof is an application of the second moment method to a certain slice of perceptron con gurations, as selected by the so-called TAP ( ouless, Anderson, and Palmer, 1977) or AMP (approximate message passing) iteration, whose scaling limit has been characterized by Bayati andMontanari (2011) and Bolthausen (2012). For verifying the condition on S‹pλq we outline one approach, which is implemented in the current version using (nonrigorous) numerical integration packages. In a future version of this paper we intend to complete the veri cation by implementing a rigorous numerical method.Krauth and Mézard [KM89] conjectured that as N Ò 8 the ratio M N pκq{N converges to an explicit constant α ‹ pκq, which for κ " 0 is roughly 0.83. is was one of several works in the statistical physics literature analyzing various perceptron models via the "replica" or "cavity" heuristics [Gar87, Gar88, GD88, Méz89]. In particular, in the variant where J ranges not over t´1,`1u N but over the entire sphere of radius N 1{2 , the analogous threshold was computed by Gardner and Derrida [GD88]. Another common variation is to take g µ,i P t´1,`1u to be i.i.d. symmetric random signs (Bernoulli disorder). e conjectured thresholds di er between the Ising [KM89] and spherical [GD88] models, but do not depend on whether the disorder is Bernoulli or gaussian. While the classical problem is under Bernoulli disorder, we have chosen to work under gaussian disorder to remove some technical di culties. For the spherical perceptron, very sharp rigorous results have been obtained, including a proof of the predicted threshold for all nonnegative κ under Bernoulli disorder [ST03]. For the Ising perceptron, much less has been proved. One can introduce a parameter β ě 0 and de ne the associated positive-temperature partition function Z κ,β (see (7) below). e replica calculation extends to a prediction [GD88, KM89] for the limit of N´1 ln Z κ,β as N Ò 8 with M {N Ñ α. is formula has been proved to be correct at su ciently high temperature (small β) under Bernoulli disorder [Tal00]. For the original model (1), corresponding to zero temperature or β " 8, the best rigorous result to date is that for κ " 0 there exists positive γ such that γ ď M N {N ď 1´γ with high probability [KR98, Tal99] as N Ò 8 (see also [Sto13] for some work on general κ). Our m...

Storage capacity in symmetric binary perceptrons

“…Up to very recently only widely non-matching upper bounds and lower bounds for the storage capacity of the binary perceptron were available 13,14 . As the present work was being finalized Ding and Sun 15 proved in a remarkable paper a lower bound on the capacity that matches the Krauth and Mezard conjecture (note that much like Theorem 4 below, the main theorem in 15 depends on a numerical hypothesis). A matching upper bound remains an open challenge in mathematical physics and probability theory.…”

“…Under our definition of α r c (K) and α u c (K), we must prove two statements to show that α r c (K) = − log(2)/ log(p r,K ) (and similarly for α u c (K)). We use the first moment method to show that for α > − log(2)/ log(p r,K ), lim N →∞ Pr(E r (N, M )) = 0; then we use the second moment method to show that for α < − log(2)/ log(p r,K ), lim inf N →∞ Pr(E r (N, M )) > 0 (a result analogous to what Ding and Sun prove for the more challenging step binary perceptron 15 ). Conjecture 2 asserts the stronger statement that for α < − log(2)/ log(p r,K ), lim N →∞ Pr(E r (N, M )) = 1.…”

“…(20) Φ 1RSB {q 0 = q RS , q 1 = q RS } = Φ 1RSB {q 0 = q RS , q 1 = 1}, and the complexity of the f1RSB solution equals the RS entropy Σ(φ = 0) = φ RS eq. (22,15). However, RS and f1RSB do not share the same configuration space.…”

Shaping the learning landscape in neural networks around wide flat minima

“…This result has been put recently on rigorous grounds by ref. [7]. Similar calculations predict that for any α ∈ (0, α c ), the vast majority of the exponentially numerous solutions on the hypercube W ∈ {−1, 1} N are isolated, separated by a O (N ) Hamming mutual distance [8].…”

“…[6] (but see also the rigorous bounds in ref. [7]), perfect classification is possible with probability 1 in the limit of large N up to a critical value of α, usually denoted as α c ; above this value, the probability of finding a solution drops to zero. α c is called the capacity of the device.…”