Bagging is a small-data-set phenomenon

Chawla, Nitesh V.; Moore, Thomas E.; Bowyer, Kevin W.; Hall, Lawrence O.; Springer, Clayton; Kegelmeyer, Philip

doi:10.1109/cvpr.2001.991030

Cited by 6 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Results obtained here seem to support the position that bagging results depend simply on obtaining a diverse set of classifiers Breiman [1996], Chawla et al [2001], Dietterich [2000], Domingos [1996]. Building classifiers on disjoint partitions of the data provides a set of classifiers that meet this requirement.…”

Section: Conclusion and Discussionsupporting

confidence: 50%

Distributed learning with bagging-like performance

Chawla

Moore

Hall

et al. 2003

Pattern Recognition Letters

View full text Add to dashboard Cite

show abstract

Section: Conclusion and Discussionsupporting

confidence: 50%

Distributed learning with bagging-like performance

Chawla

Moore

Hall

et al. 2003

Pattern Recognition Letters

View full text Add to dashboard Cite

show abstract

“…With regard to the former issue, currently the main explanation of bagging operation is given in terms of its capability to reduce the variance component of the misclassification probability, which was related by Breiman [3] to the degree of "instability" of the base classifier, informally defined as the tendency of undergoing large changes in its decision function as a result of small changes in the training set: the more unstable a classifier, the higher the variance component of its misclassification probability and thus the improvement attained by bagging. For classification problems, this DRAFT explanation is supported by empirical evidence [2], [6], [16], [20], according to several biasvariance decompositions proposed so far, although alternative explanations have been proposed as well (for instance [5], [7], [10]), and some works showed that bagging can also reduce bias [2], [20]. With regard to the latter issue above, it is well known that bagging miclassification rate tends to an asymptotic value as the ensemble size increases.…”

Section: Introductionmentioning

confidence: 77%

“…We point out however that this last conclusion can not be formally derived from the results of Sect. II-B, since the bias-variance decomposition of bagging error given by ( 8) is related to the error of a single classifier trained not on the original training set (1), but on a bootstrap replicate of it (5). For the same reason, the results of Sect.…”

Section: Discussionmentioning

confidence: 96%

“…1): the higher the variance of x b , the more unstable the classifier. Accordingly, the variance component of the expected DRAFT added error (1) of an unstable classifier will be high, as well as the one of a classifier trained on a bootstrap replicate of the original training set (5). Instead, the variance component of the error of m > 1 bagged classifiers will be smaller than that of a single bootstrap replicate, according to (8), and can be expected to be smaller than that of a classifier trained on the original training set (1) as well.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

A Theoretical Analysis of Bagging as a Linear Combination of Classifiers

Fumera

Roli

Serrau

2008

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We apply an analytical framework for the analysis of linearly combined classifiers to ensembles generated by bagging. This provides an analytical model of bagging misclassification probability as a function of the ensemble size, which is a novel result in the literature. Experimental results on real data sets confirm the theoretical predictions. This allows us to derive a novel and theoretically grounded guideline for choosing bagging ensemble size. Furthermore, our results are consistent with explanations of bagging in terms of classifier instability and variance reduction, support the optimality of the simple average over the weighted average combining rule for ensembles generated by bagging, and apply to other randomization-based methods for constructing classifier ensembles. Although our results do not allow to compare bagging misclassification probability with the one of an individual classifier trained on the original training set, we discuss how the considered theoretical framework could be exploited to this aim.

show abstract

“…More or less diversity in an ensemble is achieved depending on the level of dissimilarity between the training sets of the base classifiers composing the ensemble. However, bootstrapped training sets become more and more similar as redundancy is increasing (Chawla et al, ). Redundancy not only slows down the training task but it can significantly decrease diversity in bagging thus degrading the performance of bagging, affecting the rarer and harder classes.…”

Section: Introductionmentioning

confidence: 99%

Building bagging on critical instances

2019

View full text Add to dashboard Cite

The ensemble method is a powerful data mining paradigm, which builds a classification model by integrating multiple diversified component learners. Bagging is one of the most successful ensemble methods. It is made of bootstrap‐inspired classifiers and uses these classifiers to get an aggregated classifier. However, in bagging, bootstrapped training sets become more and more similar as redundancy is increasing. Besides redundancy, any training set is usually subject to noise. Moreover, the training set might be imbalanced. Thus, each training instance has a different impact on the learning process. This paper explores some properties of the ensemble margin and its use in improving the performance of bagging. We introduce a new approach to measure the importance of training data in learning, based on the margin theory. Then, a new bagging method concentrating on critical instances is proposed. This method is more accurate than bagging and more robust than boosting. Compared to bagging, it reduces the bias while generally keeping the same variance. Our findings suggest that (a) examples with low margins tend to be more critical for the classifier performance; (b) examples with higher margins tend to be more redundant; (c) misclassified examples with high margins tend to be noisy examples. Our experimental results on 15 various data sets show that the generalization error of bagging can be reduced up to 2.5% and its resilience to noise strengthened by iteratively removing both typical and noisy training instances, reducing the training set size by up to 75%.

show abstract

Bagging is a small-data-set phenomenon

Cited by 6 publications

References 7 publications

Distributed learning with bagging-like performance

Distributed learning with bagging-like performance

A Theoretical Analysis of Bagging as a Linear Combination of Classifiers

Building bagging on critical instances

Contact Info

Product

Resources

About