Empirical Margin Distributions and Bounding the Generalization Error of Combined Classifiers

Koltchinskii, Vladimir; Panchenko, Dmitry

doi:10.1214/aos/1015362183

Cited by 310 publications

(259 citation statements)

References 31 publications

Supporting

Mentioning

253

Contrasting

Unclassified

Order By: Relevance

“…Together with Theorem 3.2, this result establishes a precise characterization of the computational and statistical behavior of AdaBoost for all iterations above the threshold T (n), and notably complements the classical margin upper bounds [81,53]. Thus, Theorem 3.4 reinforces a crucial conclusion from Section 3.1-the max-1 -margin is the key quantity governing the generalization error of AdaBoost, and as emphasized earlier, this resolves a series of discussions around this topic [81,15].…”

Section: Boosting In High Dimensionssupporting

confidence: 66%

“…• Turning to boosting, we provide a sharp characterization of a threshold T such that for all iterations t ≥ T , the AdaBoost iterates (with a properly scaled step size) stay arbitrarily close toθ n, 1 , in the large n, p limit (1.1) (Theorem 3.4). Together with Theorems 3.1-3.2, this result immediately provides an exact characterization of the generalization error of boosting, and improves upon the existing upper bounds [81,53] by a margin. This formula crucially involves κ , and therefore, our results resolve an open question posed in Schapire et al [81], Breiman [15] regarding which quantity truly governs the generalization performance of AdaBoost.…”

Section: Introductionsupporting

confidence: 71%

“…Since our result is asymptotically sharp (rather than merely an upper bound), Theorem 3.2 provides an answer to the question posed in [81,15]. In contrast, existing margin-based generalization upper bounds [81,53] scale as…”

Section: Theorem 32 Under the Assumptions Of Theorem 31 The Generalization Error Of Any Min-mentioning

confidence: 75%

“…Breiman [15] subsequently contended these empirical margin distribution based explanations, using extensive simulations, and proposed to bound the generalization error using the minimum value of the margin over the training set. Later, Koltchinskii and Panchenko [53] improved the earlier bounds from [81]. Since all of these results involved upper bounds, a proper understanding of AdaBoost was still far from achieved; in fact Breiman in his Wald lectures (2002) commented that he would characterize "the understanding of Adaboost as the most important open problem in machine learning."…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-L1-Norm Interpolated Classifiers

Liang

Sur²

2020

SSRN Journal

View full text Add to dashboard Cite

This paper establishes a precise high-dimensional asymptotic theory for boosting on separable data, taking statistical and computational perspectives. We consider the setting where the number of features (weak learners) p scales with the sample size n, in an over-parametrized regime. Under a broad class of statistical models, we provide an exact analysis of the generalization error of boosting, when the algorithm interpolates the training data and maximizes the empirical 1 -margin. The relation between the boosting test error and the optimal Bayes error is pinned down explicitly. In turn, these precise characterizations resolve several open questions raised in [15,81] surrounding boosting. On the computational front, we provide a sharp analysis of the stopping time when boosting approximately maximizes the empirical 1 margin. Furthermore, we discover that the larger the overparametrization ratio p/n, the smaller the proportion of active features (with zero initialization), and the faster the optimization reaches interpolation. At the heart of our theory lies an in-depth study of the maximum 1 -margin, which can be accurately described by a new system of non-linear equations; we analyze this margin and the properties of this system, using Gaussian comparison techniques and a novel uniform deviation argument. Variants of AdaBoost corresponding to general q geometry, for q > 1, are also presented, together with an exact analysis of the high-dimensional generalization and optimization behavior of a class of these algorithms.

show abstract

Section: Boosting In High Dimensionssupporting

confidence: 66%

Section: Introductionsupporting

confidence: 71%

Section: Theorem 32 Under the Assumptions Of Theorem 31 The Generalization Error Of Any Min-mentioning

confidence: 75%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-L1-Norm Interpolated Classifiers

Liang

Sur²

2020

SSRN Journal

View full text Add to dashboard Cite

show abstract

“…This section analyzes the generalization error of the proposed CoUDA method based on Rademacher complexity [24], [56], [34], [57]. Before that, we give some necessary notations.…”

Section: Theoretical Analysismentioning

confidence: 99%

Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis

Zhang

Wei

et al. 2020

IEEE Trans. on Image Process.

140

View full text Add to dashboard Cite

Deep learning based medical image diagnosis has shown great potential in clinical medicine. However, it often suffers two major difficulties in real-world applications: 1) only limited labels are available for model training, due to expensive annotation costs over medical images; 2) labeled images may contain considerable label noise (e.g., mislabeling labels) due to diagnostic difficulties of diseases. To address these, we seek to exploit rich labeled data from relevant domains to help the learning in the target task via Unsupervised Domain Adaptation (UDA). Unlike most UDA methods that rely on clean labeled data or assume samples are equally transferable, we innovatively propose a Collaborative Unsupervised Domain Adaptation algorithm, which conducts transferability-aware adaptation and conquers label noise in a collaborative way. We theoretically analyze the generalization performance of the proposed method, and also empirically evaluate it on both medical and general images. Promising experimental results demonstrate the superiority and generalization of the proposed method.

show abstract

Embedding with a Lipschitz function

Mendelson¹

2005

Random Struct Algorithms

View full text Add to dashboard Cite

ABSTRACT:We investigate a new notion of embedding of subsets of {−1, 1} n in a given normed space, in a way which preserves the structure of the given set as a class of functions on {1, . . . , n}. This notion is an extension of the margin parameter often used in Nonparametric Statistics. Our main result is that even when considering "small" subsets of {−1, 1} n , the vast majority of such sets do not embed in a better way than the entire cube in any normed space that satisfies a minor structural assumption.

show abstract

Empirical Margin Distributions and Bounding the Generalization Error of Combined Classifiers

Cited by 310 publications

References 31 publications

A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-L1-Norm Interpolated Classifiers

A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-L1-Norm Interpolated Classifiers

Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis

Embedding with a Lipschitz function

Contact Info

Product

Resources

About