Towards an Understanding of Benign Overfitting in Neural Networks

Li, Zhu; Zhou, Zhi-Hua; Gretton, Arthur

doi:10.48550/arxiv.2106.03212

Cited by 5 publications

(7 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The assumption that we have access to the exact form of the singular value decomposition of the feature matrix is only made trivialize the calculations, and is by no means essential. When the optimization algorithm is stopped before perfect matching can be achieve, the fitting from the weighted optimization scheme generated a smoother approximation (with a smaller p, according to (23), than what we would obtain with a regular least-squares minimization).…”

Section: Extension To General Feature Regressionmentioning

confidence: 98%

“…In this section, we study the more realistic setting of training with noisy data. The common practice is that when we solve the minimization problem in such a case, we should avoid overfitting the model to the data by stopping the optimization algorithm early at an appropriate level of values depending on the noise level in the data, but see [3,23] for some analysis in the direction of "benign overfitting". When training with noisy data, the impact of the weight matrices Λ −α…”

Section: Error Bounds For Training With Noisy Datamentioning

confidence: 99%

See 1 more Smart Citation

A Generalized Weighted Optimization Method for Computational Learning and Inversion

Engquist¹,

Ren²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

The generalization capacity of various machine learning models exhibits different phenomena in the underand over-parameterized regimes. In this paper, we focus on regression models such as feature regression and kernel regression and analyze a generalized weighted least-squares optimization method for computational learning and inversion with noisy data. The highlight of the proposed framework is that we allow weighting in both the parameter space and the data space. The weighting scheme encodes both a priori knowledge on the object to be learned and a strategy to weight the contribution of different data points in the loss function. Here, we characterize the impact of the weighting scheme on the generalization error of the learning method, where we derive explicit generalization errors for the random Fourier feature model in both the under-and over-parameterized regimes. For more general feature maps, error bounds are provided based on the singular values of the feature matrix. We demonstrate that appropriate weighting from prior knowledge can improve the generalization capability of the learned model.

show abstract

Section: Extension To General Feature Regressionmentioning

confidence: 98%

Section: Error Bounds For Training With Noisy Datamentioning

confidence: 99%

A Generalized Weighted Optimization Method for Computational Learning and Inversion

Engquist¹,

Ren²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While they offered important insights into the benign overfitting phenomenon, most of them are limited to the settings of linear models (Belkin et al, 2019b;Bartlett et al, 2020;Hastie et al, 2019;Wu and Xu, 2020;Chatterji and Long, 2020;Zou et al, 2021b;Cao et al, 2021) and kernel/random features models (Belkin et al, 2018;Liang and Rakhlin, 2020;Montanari and Zhong, 2020), and cannot be applied to neural network models that are of greater interest. The only notable exceptions are (Adlam and Pennington, 2020;Li et al, 2021), which attempted to understand benign overfitting in neural network models. However, they are still limited to the "neural tagent kernel regime" (Jacot et al, 2018) where the neural network learning problem is essentially equivalent to kernel regression.…”

Section: Introductionmentioning

confidence: 99%

Benign Overfitting in Two-layer Convolutional Neural Networks

Chen¹,

Chen²,

Belkin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modern neural networks often have great expressive power and can be trained to overfit the training data, while still achieving a good test performance. This phenomenon is referred to as "benign overfitting". Recently, there emerges a line of works studying "benign overfitting" from the theoretical perspective. However, they are limited to linear models or kernel/random feature models, and there is still a lack of theoretical understanding about when and how benign overfitting occurs in neural networks. In this paper, we study the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN). We show that when the signal-tonoise ratio satisfies a certain condition, a two-layer CNN trained by gradient descent can achieve arbitrarily small training and test loss. On the other hand, when this condition does not hold, overfitting becomes harmful and the obtained CNN can only achieve constant level test loss. These together demonstrate a sharp phase transition between benign overfitting and harmful overfitting, driven by the signal-to-noise ratio. To the best of our knowledge, this is the first work that precisely characterizes the conditions under which benign overfitting can occur in training convolutional neural networks.

show abstract

“…Contrary to conventional statistical wisdom, overparameterization turns out to be a rather desirable property for neural networks. For instance, phenomena such as double descent [BHMM19, SGd + 19, NKB + 20] and benign overfitting [BLLT20, LZG21,BMR21] suggest that understanding the generalization properties of overparameterized models lies beyond the scope of the usual control of capacity via the size of the parameter set [NTS15].…”

Section: Introductionmentioning

confidence: 99%

Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks

Шевченко¹,

Kungurtsev²,

Mondelli³

2021

Preprint

View full text Add to dashboard Cite

Understanding the properties of neural networks trained via stochastic gradient descent (SGD) is at the heart of the theory of deep learning. In this work, we take a mean-field view, and consider a two-layer ReLU network trained via SGD for a univariate regularized regression problem. Our main result is that SGD is biased towards a simple solution: at convergence, the ReLU network implements a piecewise linear map of the inputs, and the number of "knot" points -i.e., points where the tangent of the ReLU network estimator changes -between two consecutive training inputs is at most three. In particular, as the number of neurons of the network grows, the SGD dynamics is captured by the solution of a gradient flow and, at convergence, the distribution of the weights approaches the unique minimizer of a related free energy, which has a Gibbs form. Our key technical contribution consists in the analysis of the estimator resulting from this minimizer: we show that its second derivative vanishes everywhere, except at some specific locations which represent the "knot" points. We also provide empirical evidence that knots at locations distinct from the data points might occur, as predicted by our theory.

show abstract

Towards an Understanding of Benign Overfitting in Neural Networks

Cited by 5 publications

References 24 publications

A Generalized Weighted Optimization Method for Computational Learning and Inversion

A Generalized Weighted Optimization Method for Computational Learning and Inversion

Benign Overfitting in Two-layer Convolutional Neural Networks

Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks

Contact Info

Product

Resources

About