In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Neyshabur, Behnam; Tomioka, Ryota; Srebro, Nathan

doi:10.48550/arxiv.1412.6614

Cited by 128 publications

(188 citation statements)

References 6 publications

Supporting

Mentioning

178

Contrasting

Order By: Relevance

“…An intriguing empirical phenomenon called "double descent" has recently emerged in the study of overparameterized learning models (Neyshabur et al, 2014;Nakkiran et al, 2019;Belkin et al, 2018Belkin et al, , 2019Belkin, 2021). Consider, for example, a risk curve that depicts how the generalization error varies as more parameters are added to the model.…”

Section: Motivation: a Multi-descent Phenomenonmentioning

confidence: 99%

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Li¹,

Wei²

2021

Preprint

View full text Add to dashboard Cite

An evolving line of machine learning works observe empirical evidence that suggests interpolating estimators -the ones that achieve zero training error -may not necessarily be harmful. This paper pursues theoretical understanding for an important type of interpolators: the minimum 1-norm interpolator, which is motivated by the observation that several learning algorithms favor low 1-norm solutions in the over-parameterized regime. Concretely, we consider the noisy sparse regression model under Gaussian design, focusing on linear sparsity and high-dimensional asymptotics (so that both the number of features and the sparsity level scale proportionally with the sample size).We observe, and provide rigorous theoretical justification for, a curious multi-descent phenomenon; that is, the generalization risk of the minimum 1-norm interpolator undergoes multiple (and possibly more than two) phases of descent and ascent as one increases the model capacity. This phenomenon stems from the special structure of the minimum 1-norm interpolator as well as the delicate interplay between the over-parameterized ratio and the sparsity, thus unveiling a fundamental distinction in geometry from the minimum 2-norm interpolator. Our finding is built upon an exact characterization of the risk behavior, which is governed by a system of two non-linear equations with two unknowns.

show abstract

Section: Motivation: a Multi-descent Phenomenonmentioning

confidence: 99%

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Li¹,

Wei²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Ma et al (2021) show that even without explicit regularization, when the learning rate of GD is infinitesimal, i.e., GD approximating gradient flow, X and Y maintain the gap in their magnitudes. Such an effect is more broadly recognized as implicit bias of learning algorithms (Neyshabur et al, 2014;Gunasekar et al, 2018;Soudry et al, 2018). Built upon this implicit bias, Du et al (2018) further prove that GD with diminishing learning rates converges to a bounded global minimum of (1), and this conclusion is recently extended to the case of a constant small learning rate (Ye & Du, 2021).…”

Section: Background and Related Workmentioning

confidence: 99%

Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect

Wang¹,

Chen²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent empirical advances show that training deep models with large learning rate often improves generalization performance. However, theoretical justifications on the benefits of large learning rate are highly limited, due to challenges in analysis. In this paper, we consider using Gradient Descent (GD) with a large learning rate on a homogeneous matrix factorization problem, i.e., minX,Y A − XY 2 F . We prove a convergence theory for constant large learning rates well beyond 2/L, where L is the largest eigenvalue of Hessian at the initialization. Moreover, we rigorously establish an implicit bias of GD induced by such a large learning rate, termed 'balancing', meaning that magnitudes of X and Y at the limit of GD iterations will be close even if their initialization is significantly unbalanced. Numerical experiments are provided to support our theory.

show abstract

“…By construction, the empirical risk is always non-negative, and hence so is the training error: min θ∈Θ R n (θ) ≥ 0. However, state-of-the-art models in machine learning are often overparametrized, in the sense that they can achieve vanishing, or nearly vanishing training error, even with noisy labels: min θ∈Θ R n (θ) = 0 [NTS14]. The emphasis on noise is crucial here: classically one would expect vanishing empirical risk only in the absence of noise, i.e., if we had E{ (y; f * (x))} = 0 for some function f * that happens to be in the model class {f ( •; θ)} θ∈Θ .…”

Section: Introductionmentioning

confidence: 99%

Tractability from overparametrization: The example of the negative perceptron

Montanari¹,

Zhong²,

Zhou³

2021

Preprint

View full text Add to dashboard Cite

In the negative perceptron problem we are given n data points (xi, yi), where xi is a d-dimensional vector and yi ∈ {+1, −1} is a binary label. The data are not linearly separable and hence we content ourselves to find a linear classifier with the largest possible negative margin. In other words, we want to find a unit norm vector θ that maximizes min i≤n yi θ, xi . This is a non-convex optimization problem (it is equivalent to finding a maximum norm vector in a polytope), and we study its typical properties under two random models for the data.We consider the proportional asymptotics in which n, d → ∞ with n/d → δ, and prove upper and lower bounds on the maximum margin κs(δ) or -equivalently-on its inverse function δs(κ). In other words, δs(κ) is the overparametrization threshold: for n/d ≤ δs(κ) − ε a classifier achieving vanishing training error exists with high probability, while for n/d ≥ δs(κ) + ε it does not. Our bounds on δs(κ) match to the leading order as κ → −∞. We then analyze a linear programming algorithm to find a solution, and characterize the corresponding threshold δ lin (κ). We observe a gap between the interpolation threshold δs(κ) and the linear programming threshold δ lin (κ), raising the question of the behavior of other algorithms.

show abstract

In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Cited by 128 publications

References 6 publications

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect

Tractability from overparametrization: The example of the negative perceptron

Contact Info

Product

Resources

About