Unveiling the structure of wide flat minima in neural networks

Baldassi, Carlo; Lauditi, Clarissa; Malatesta, Enrico M.; Perugini, Gabriele; Zecchina, Riccardo

doi:10.48550/arxiv.2107.01163

Cited by 2 publications

(6 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3. The overall picture closely resembles that of simpler models [19], and we point out some noteworthy results:…”

Section: 𝜙 = Limsupporting

confidence: 69%

“…Phase diagram. Before diving into analytical details, we anticipate how the geometry of the space of solutions changes < l a t e x i t s h a 1 _ b a s e 6 4 = " w z For 𝛼 𝑇 → 0 we recover the critical capacity [18] and the local entropy transition [19] of the standard binary perceptron. In the inset we show the training error of SA, BP and SBPI versus the degree of overparameterization 1/𝛼 for 𝐷 = 201 and 𝛼 𝑇 = 3.…”

Section: 𝜙 = Limmentioning

confidence: 99%

“…Here, following ref. [19], we are chiefly interested in the behavior of this quantity when 𝜅 = 0. For a given value of 𝛼 < 𝛼 𝑐 , the local entropy curves of the references exhibit different characteristics as κ varies.…”

Section: 𝜙 = Limmentioning

confidence: 99%

“…The first one is based on the use of a large deviations analysis [5,6] which however leads to a quite heavy formalism for the models under study. We have thus adopted a recently introduced simpler method [19] which gives similar results to the large deviations approach. It is based on the observation that, by definition of 𝛼 LE , references located in the large-scale dense region should not exist anymore when 𝛼 > 𝛼 LE .…”

Section: 𝜙 = Limmentioning

confidence: 99%

See 3 more Smart Citations

Learning through atypical "phase transitions" in overparameterized neural networks

Baldassi,

Lauditi,

Malatesta

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Current deep neural networks are highly overparameterized (up to billions of connection weights) and nonlinear. Yet they can fit data almost perfectly through variants of gradient descent algorithms and achieve unexpected levels of prediction accuracy without overfitting. These are formidable results that escape the bias-variance predictions of statistical learning and pose conceptual challenges for non-convex optimization. In this paper, we use methods from statistical physics of disordered systems to analytically study the computational fallout of overparameterization in nonconvex neural network models. As the number of connection weights increases, we follow the changes of the geometrical structure of different minima of the error loss function and relate them to learning and generalisation performance. We find that there exist a gap between the SAT/UNSAT interpolation transition where solutions begin to exist and the point where algorithms start to find solutions, i.e. where accessible solutions appear. This second phase transition coincides with the discontinuous appearance of atypical solutions that are locally extremely entropic, i.e., flat regions of the weight space that are particularly solution-dense and have good generalization properties. Although exponentially rare compared to typical solutions (which are narrower and extremely difficult to sample), entropic solutions are accessible to the algorithms used in learning. We can characterize the generalization error of different solutions and optimize the Bayesian prediction, for data generated from a structurally different network. Numerical tests on observables suggested by the theory confirm that the scenario extends to realistic deep networks.

show abstract

“…3. The overall picture closely resembles that of simpler models [19], and we point out some noteworthy results:…”

Section: 𝜙 = Limsupporting

confidence: 69%

Section: 𝜙 = Limmentioning

confidence: 99%

Section: 𝜙 = Limmentioning

confidence: 99%

Section: 𝜙 = Limmentioning

confidence: 99%

See 2 more Smart Citations

Learning through atypical "phase transitions" in overparameterized neural networks

Baldassi,

Lauditi,

Malatesta

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This phenomenon has been justified numerically by the existence of subdominant and dense connected regions of solutions [Bal+15]. We also refer to [Bal+21] for heuristic descriptions of the solution space.…”

Section: Introductionmentioning

confidence: 97%

Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster

Abbé¹,

Li²,

Sly³

2021

Preprint

View full text Add to dashboard Cite

It was recently shown that almost all solutions in the symmetric binary perceptron are isolated, even at low constraint densities, suggesting that finding typical solutions is hard. In contrast, some algorithms have been shown empirically to succeed in finding solutions at low density. This phenomenon has been justified numerically by the existence of subdominant and dense connected regions of solutions, which are accessible by simple learning algorithms. In this paper, we establish formally such a phenomenon for both the symmetric and asymmetric binary perceptrons. We show that at low constraint density (equivalently for overparametrized perceptrons), there exists indeed a subdominant connected cluster of solutions with almost maximal diameter, and that an efficient multiscale majority algorithm can find solutions in such a cluster with high probability, settling in particular an open problem posed by Perkins-Xu in STOC'21. In addition, even close to the critical threshold, we show that there exist clusters of linear diameter for the symmetric perceptron, as well as for the asymmetric perceptron under additional assumptions.

show abstract

Unveiling the structure of wide flat minima in neural networks

Cited by 2 publications

References 22 publications

Learning through atypical "phase transitions" in overparameterized neural networks

Learning through atypical "phase transitions" in overparameterized neural networks

Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster

Contact Info

Product

Resources

About