Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II

Abstract

We study the optimization problem associated with fitting two-layer ReLU neural networks with respect to the squared loss, where labels are generated by a target network. We make use of the rich symmetry structure to develop a novel set of tools for studying families of spurious minima. In contrast to existing approaches which operate in limiting regimes, our technique directly addresses the nonconvex loss landscape for a finite number of inputs d and neurons k, and provides analytic, rather than heuristic, information. In particular, we derive analytic estimates for the loss at different minima, and prove that modulo O(d −1/2 )-terms the Hessian spectrum concentrates near small positive constants, with the exception of Θ(d) eigenvalues which grow linearly with d. We further show that the Hessian spectrum at global and spurious minima coincide to O(d −1/2 )-order, thus challenging our ability to argue about statistical generalization through local curvature. Lastly, our technique provides the exact fractional dimensionality at which families of critical points turn from saddles into spurious minima. This makes possible the study of the creation and the annihilation of spurious minima using powerful tools from equivariant bifurcation theory.

Equivariant bifurcation, quadratic equivariants, and symmetry breaking for the standard representation of S k

“…The FPS result is used in [3], together with results from the representation theory of the symmetric group, to obtain precise results on the Hessian spectrum for several families of spurious minima in shallow neural networks (valid for arbitrarily large k). In [5], these results are extended to two-layer ReLU networks where it is shown that to order O(k − 1 2 ) the spectra are identical for the global minima and several families of spurious minima. All results to this point assume that the number of inputs d to the network is greater than or equal to the number of neurons k. In [6], the overparametrized case k > d is analysed and it is shown, using FPS methods and representation theory, that the addition of one or two neurons annihilates certain families of spurious minima of types I and II (as defined in [6]).…”

“…In the concluding comments, we return to the original motivating problem about the creation and annihilation of spurious minima, indicate how the results of the paper can be used to understand this phenomenon, and discussed related current and proposed developments. [1][2][3][4][5][6] Article [2] identifies spurious minima as examples of symmetry breaking in the student-teacher model and gives an extensive numerical study of the phenomenon in wide range of settings. In [4] several infinite families of critical points of spurious minima are constructed and it is shown that these critical points may be represented by convergent fractional power series (FPS) in 1/ √ k (k is the number of neurons viewed as a real parameter).…”

“…For example, if k = 17, the minimal model will have exactly 52 666 saddle-node bifurcations and 12 870 hyperbolic solution curves. The minimal model is indicative of the complex landscape geometry that is involved in the creation of asymmetric spurious minima in non-convex optimisation in neural networks (cf [3,5]). A similar result holds for k even-on account of the cubic terms in (1.2), the 'standard' model we perturb is defined slightly differently so that no solutions are introduced which are unrelated to the bifurcation.…”

“…It is easily shown [15, theorem 6.10.1] that S induces a stratification S of R m−1 -the strata of S consist of the strata of S which are subsets of R m−1 intersected, if necessary 5 , with R m−1 . We have Γ f S at (0, 0) iff γ f S at λ = 0.…”

“…This is followed by an outline of contents and description of the mathematical contributions-the focus of the article. At the suggestion of the referees, brief notes are included at the end of the introduction on the relationship of [1][2][3][4][5][6] to this paper.…”