Learning through atypical "phase transitions" in overparameterized neural networks

Abstract

Current deep neural networks are highly overparameterized (up to billions of connection weights) and nonlinear. Yet they can fit data almost perfectly through variants of gradient descent algorithms and achieve unexpected levels of prediction accuracy without overfitting. These are formidable results that escape the bias-variance predictions of statistical learning and pose conceptual challenges for non-convex optimization. In this paper, we use methods from statistical physics of disordered systems to analytically study the computational fallout of overparameterization in nonconvex neural network models. As the number of connection weights increases, we follow the changes of the geometrical structure of different minima of the error loss function and relate them to learning and generalisation performance. We find that there exist a gap between the SAT/UNSAT interpolation transition where solutions begin to exist and the point where algorithms start to find solutions, i.e. where accessible solutions appear. This second phase transition coincides with the discontinuous appearance of atypical solutions that are locally extremely entropic, i.e., flat regions of the weight space that are particularly solution-dense and have good generalization properties. Although exponentially rare compared to typical solutions (which are narrower and extremely difficult to sample), entropic solutions are accessible to the algorithms used in learning. We can characterize the generalization error of different solutions and optimize the Bayesian prediction, for data generated from a structurally different network. Numerical tests on observables suggested by the theory confirm that the scenario extends to realistic deep networks.

Deep networks on toroids: removing symmetries reveals the structure of flat regions in the landscape geometry*

“…Also, with few exceptions, the solutions we find can be connected by paths of (near-)zero error with a single bend. Overall, our results are compatible with the analysis of the geometry of the space of solutions in binary, shallow networks reported in Baldassi et al (2021a) and(2021b), according to which efficient algorithms target large connected structures of solutions with the more robust (flatter) ones at the center, radiating out into progressively sharper ones.…”

Universal mean-field upper bound for the generalization gap of deep neural networks

“…Unfortunately, it is challenging to extend this approach to deeper networks. Other groups are studying the role of overparametrisation and related phenomena such as the double descent in the regime of lazy training [14][15][16][17][18][19][20]. Also the statistical physics of kernel learning (originally started in [21]) has undergone a revival in the last few years [22], mainly due to the discovery of the Neural Tangent Kernel (NTK) limit of deep neural networks -a mathematical equivalence between neural networks and a certain kernel that arises in the limit of large layer size [23,24].…”

Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems

“…Theoretical studies of neural networks become increasingly important in recent years [1][2][3], as deep neural networks are widely used in various domains of both scientific and industrial communities. One of the most powerful theoretical tools is the replica method, which is able to derive equilibrium properties of neural networks (systems of interacting neurons or synapses), such as phase diagram [4][5][6][7], storage capacity [8][9][10][11], and even large-deviation behavior of learning algorithms [12][13][14]. Intuitively, the replica method introduces n (an integer) copies of the original system.…”

“…However, the region of the solution space accessed by practical algorithms does not belong to the equilibrium hard-to-reach isolated parts, but subdominant dense parts [12,13]. These dense parts are further shown to have good generalization properties [14,30], providing a new paradigm to understand deep learning. In this work, we show how the algorithmic instability is connected to the instability of the replica symmetric (RS) solution (or replicon mode) of the model.…”