Tractability from overparametrization: The example of the negative perceptron

Abstract

In the negative perceptron problem we are given n data points (xi, yi), where xi is a d-dimensional vector and yi ∈ {+1, −1} is a binary label. The data are not linearly separable and hence we content ourselves to find a linear classifier with the largest possible negative margin. In other words, we want to find a unit norm vector θ that maximizes min i≤n yi θ, xi . This is a non-convex optimization problem (it is equivalent to finding a maximum norm vector in a polytope), and we study its typical properties under two random models for the data.We consider the proportional asymptotics in which n, d → ∞ with n/d → δ, and prove upper and lower bounds on the maximum margin κs(δ) or -equivalently-on its inverse function δs(κ). In other words, δs(κ) is the overparametrization threshold: for n/d ≤ δs(κ) − ε a classifier achieving vanishing training error exists with high probability, while for n/d ≥ δs(κ) + ε it does not. Our bounds on δs(κ) match to the leading order as κ → −∞. We then analyze a linear programming algorithm to find a solution, and characterize the corresponding threshold δ lin (κ). We observe a gap between the interpolation threshold δs(κ) and the linear programming threshold δ lin (κ), raising the question of the behavior of other algorithms.

Injectivity of ReLU networks: perspectives from statistical physics

“…to soft sphere packing), studies of the spherical perceptron in physics and mathematics are numerous. Without aiming at being exhaustive, and rather primarily referring to works relevant for our presentation, these studies include [GD88, Gar88, FPS + 17] in the physics literature, while the spherical perceptron has also been studied with mathematically rigorous techniques [ST02], [Tal10, Chapter 3], [Tal11, Chapter 8], [Sto13a,Sto13b,MZZ21]. In particular, the satisfiability threshold α = 2 has been rigorously determined.…”

“…In this model, even computing the Gardner capacity conjecturally requires the full-RSB prediction. However, [Sto13b,MZZ21] have made a refined use of Gordon's inequality to improve over the replica-symmetric upper bound for the capacity. While similar ideas might be able to improve the upper bound of Theorem 1.8, it is not immediate to implement them, since the method used in [MZZ21] relies on the min-max problem being formulated over unit-norm vectors, which is not the case here.…”

“…However, [Sto13b,MZZ21] have made a refined use of Gordon's inequality to improve over the replica-symmetric upper bound for the capacity. While similar ideas might be able to improve the upper bound of Theorem 1.8, it is not immediate to implement them, since the method used in [MZZ21] relies on the min-max problem being formulated over unit-norm vectors, which is not the case here. Since Theorem 1.8 already allows to disprove the Euler characteristic prediction, we leave such an improvement for later work.…”

Gaussian Universality of Linear Classifiers with Random Labels in High-Dimension

“…In fact, the capacity of generalized linear classi ers to t random labels for the simple case of Gaussian generated data, is also a classical topic, e.g. [16][17][18][19][20][21][22][23][24].…”

Sharp threshold sequence and universality for Ising perceptron models

“…As mentioned earlier, the perceptron model was analyzed in the (nonrigorous) statistical physics literature in the 1980s [Gar88,GD88,KM89,Méz89]. We refer to [Cov65,Gar88,MRSY19,MZZ21] for discussions of statistical motivations for the model.…”