Noise peeling methods to improve boosting algorithms

Martinez, Waldyn G.; Gray, Jeff

doi:10.1016/j.csda.2015.06.010

Cited by 12 publications

(5 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although bagging and random forests generally perform better than AdaBoost under noisy circumstances, they are still not completely robust to noise and outliers [Maclin and Opitz, 1997]. Deleting outliers (also called noise filtering or noise peeling) by pre-processing the data is preferable under certain high noise circumstances [Martinez and Gray, 2016].…”

Section: Qmm Performance Under Noisementioning

confidence: 99%

Ensemble Pruning via Quadratic Margin Maximization

Martinez

2021

IEEE Access

View full text Add to dashboard Cite

Ensemble models refer to methods that combine a typically large number of classifiers into a compound prediction. The output of an ensemble method is the result of fitting a base-learning algorithm to a given data set, and obtaining diverse answers by reweighting the observations or by resampling them using a given probabilistic selection. A key challenge of using ensembles in large-scale multidimensional data lies in the complexity and the computational burden associated with them. The models created by ensembles are often difficult, if not impossible, to interpret and their implementation requires more computational power than single classifiers. Recent research effort in the field has concentrated in reducing ensemble size, while maintaining their predictive accuracy. We propose a method to prune an ensemble solution by optimizing its margin distribution, while increasing its diversity. The proposed algorithm results in an ensemble that uses only a fraction of the original classifiers, with improved or similar generalization performance. We analyze and test our method on both synthetic and real data sets. The simulations show that the proposed method compares favorably to the original ensemble solutions and to other existing ensemble pruning methodologies.

show abstract

Section: Qmm Performance Under Noisementioning

confidence: 99%

Ensemble Pruning via Quadratic Margin Maximization

Martinez

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Boosting ensemble is another common metal algorithm. The algorithm was developed from the Probably Approximately Correct (PAC) framework coined by Kearns and Valiant in an effort to enhance the accuracy of classifications [59]. Essentially, boosting was developed in an effort to convert the weak learners into strong learners to bolster their overall classification accuracy [44].…”

Section: Boostingmentioning

confidence: 99%

The Superiority of the Ensemble Classification Methods: A Comprehensive Review

Nzuva¹,

Nderu²

2019

JIEA

View full text Add to dashboard Cite

The modern technologies, which are characterized by cyber-physical systems and internet of things expose organizations to big data, which in turn can be processed to derive actionable knowledge. Machine learning techniques have vastly been employed in both supervised and unsupervised environments in an effort to develop systems that are capable of making feasible decisions in light of past data. In order to enhance the accuracy of supervised learning algorithms, various classification-based ensemble methods have been developed. Herein, we review the superiority exhibited by ensemble learning algorithms based on the past that has been carried out over the years. Moreover, we proceed to compare and discuss the common classification-based ensemble methods, with an emphasis on the boosting and bagging ensemble-learning models. We conclude by out setting the superiority of the ensemble learning models over individual base learners.

show abstract

“…The bound in (4) suggests the generalization error would degrade as the number of rounds of boosting T increased. This in fact happens, especially in cases where there is noise, (see, e.g., Grove and Schuurmans 1998;Dietterich 2000;Long and Servedio 2010;Martinez and Gray 2016), but under certain conditions boosting methods have shown not to overfit even when thousands of rounds are run (see, e.g., Breiman 1996b;Drucker and Cortes 1996;Quinlan 1996;Grove and Schuurmans 1998;Opitz and Maclin 1999;Buja 2000;Lugosi and Vayatis 2004). To better explain the effectiveness of AdaBoost and other ensembles, Schapire et al (1998) developed an upper bound on the generalization error of any ensemble method, based on the margins of the training data, from which it was concluded that larger margins should lead to a lower generalization error of the ensemble, everything else being equal.…”

Section: Generalization Error Bounds Based On Marginsmentioning

confidence: 99%

On the Insufficiency of the Large Margins Theory in Explaining the Performance of Ensemble Methods

Martinez,

Gray

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Boosting and other ensemble methods combine a large number of weak classifiers through weighted voting to produce stronger predictive models. To explain the successful performance of boosting algorithms, Schapire et al. (1998) showed that AdaBoost is especially effective at increasing the margins of the training data. Schapire et al. (1998) also developed an upper bound on the generalization error of any ensemble based on the margins of the training data, from which it was concluded that larger margins should lead to lower generalization error, everything else being equal (sometimes referred to as the "large margins theory"). Tighter bounds have been derived and have reinforced the large margins theory hypothesis. For instance, Wang et al. (2011) suggest that specific margin instances, such as the equilibrium margin, can better summarize the margins distribution. These results have led many researchers to consider direct optimization of the margins to improve ensemble generalization error with mixed results. We show that the large margins theory is not sufficient for explaining the performance of voting classifiers. We do this by illustrating how it is possible to improve upon the margin distribution of an ensemble solution, while keeping the complexity fixed, yet not improve the test set performance.

show abstract

Noise peeling methods to improve boosting algorithms

Cited by 12 publications

References 42 publications

Ensemble Pruning via Quadratic Margin Maximization

Ensemble Pruning via Quadratic Margin Maximization

The Superiority of the Ensemble Classification Methods: A Comprehensive Review

On the Insufficiency of the Large Margins Theory in Explaining the Performance of Ensemble Methods

Contact Info

Product

Resources

About