Reconciling modern machine-learning practice and the classical bias–variance trade-off

Abstract

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners.In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double descent" curve subsumes the textbook U-shaped biasvariance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.

Harmless Interpolation of Noisy Data in Regression

“…We take n = 10 noisy measurements of this signal. Example 5 is directly inspired by feature models 17 in several recent papers [7,8,52], as well as having philosophical connections to other recent papers [6,46]. Figure 3 shows the performance of the minimum-2 -norm interpolator on Example 5 as we increase the number of features.…”

“…Most recently, a double-descent curve on the test error (0 − 1 loss and MSE) as a function of the number of parameters of several parametric models was observed on several common datasets by physicists [37] and machine learning researchers [38] respectively. In these experiments, the minimum 2 -norm interpolating solution is used, and several feature families, including kernel approximators [39], were considered.…”

“…Since the publication 8 of the double descent experiments [37,38] at the end of the year 2018, there has been extremely active interest in understanding the generalization abilities of interpolating solutions in linear regression. An earlier edition of our work was presented at Information Theory and Applications, February 2019 and subsequently accepted to IEEE International Symposium on Information Theory, July 2019.…”

“…It is instructive to ask whether the above tradeoff in maximizing signal "survival" and minimizing signal "contamination" manifests as a clean bias-variance tradeoff [52]. The issue is that the contamination can arise through signal and/or noise energy.…”

Advanced Control Methods for Particle Accelerators (ACM4PA) 2019

“…symmetries, random shifts and crops, cutouts / random erasing, mixup), but note some things like mixup have not been used for regression, but in some cases worth just trying for your specific case. [49] showing that unlike traditional regression approaches, once NNs become sufficiently wide and deep they can generalize and do not overfit despite such a large number of parameters.…”