Proof of the Contiguity Conjecture and Lognormal Limit for the Symmetric Perceptron

Abstract

We consider the symmetric binary perceptron model, a simple model of neural networks that has gathered significant attention in the statistical physics, information theory and probability theory communities, with recent connections made to the performance of learning algorithms in Baldassi et al. '15.We establish that the partition function of this model, normalized by its expected value, converges to a lognormal distribution. As a consequence, this allows us to establish several conjectures for this model: (i) it proves the contiguity conjecture of Aubin et al. '19 between the planted and unplanted models in the satisfiable regime; (ii) it establishes the sharp threshold conjecture; (iii) it proves the frozen 1-RSB conjecture in the symmetric case, conjectured first by Krauth-Mézard '89 in the asymmetric case.In a recent concurrent work of Perkins-Xu [PX21], the last two conjectures were also established by proving that the partition function concentrates on an exponential scale. This left open the contiguity conjecture and the lognormal limit characterization, which are established here. In particular, our proof technique relies on a dense counter-part of the small graph conditioning method, which was developed for sparse models in the celebrated work of Robinson and Wormald.

The Overlap Gap Property: a Geometric Barrier to Optimizing over Random Structures

“…This was confirmed rigorously as an upper bound in [DS19]. For the symmetric case a similar sharp threshold α SAT (κ) is known rigorously as a κ dependent constant α(κ), as established in [PX21] and [ALS21].…”

“…While the algorithm has not been adopted to the symmetric case, it is quite likely that a version of it should work here as well for some positive sufficiently small κ-dependent constant α. Heuristically, the message passing algorithm was found to be effective at small positive densities, as reported in [BZ06]. Curiously, though, it is known that the symmetric model exhibits the weak clustering property at all positive densities α > 0, as was rigorously verified recently in [PX21] and [ALS21]. Furthermore, quite remarkably, each cluster consist of singletons!…”

Unveiling the structure of wide flat minima in neural networks

“…In the whole SAT phase, zero-margin solutions are isolated, meaning that one has to flip an extensive number of weights in order to find the closest solution. This scenario was also recently confirmed in simple one-hidden layer neural networks with generic activation functions [15] and also rigorously for the symmetric perceptron [23,24]. This kind of landscape with point-like solutions suggests that finding such solution should be a hard optimization problem; however, this is contrary to the numerical evidence given by simple algorithms such as the ones based on message passing [25,26].…”

Gardner formula for Ising perceptron models at small densities

“…e second moment analysis was done for the cases p q " 1t| | ď u and p q " 1t| | ě u in [APZ19]. For the model p q " 1t| | ď u, much ner structural results (on the typical geometry of the solution space) were obtained by [PX21,ALS21].…”