Cost-Sensitive Boosting

Abstract

A novel framework, based on the statistical interpretation of boosting, is proposed for the design of cost sensitive boosting algorithms. It is argued that, although predictors produced with boosting converge to the ratio of posterior class probabilities that also appears in Bayes decision rule, this convergence only occurs in a small neighborhood of the optimal cost-insensitive classification boundary. This is due to a combination of the cost-insensitive nature of current boosting losses, and boosting's sample reweighing mechanism. It is then shown that convergence in the neighborhood of a target cost-sensitive boundary can be achieved through boosting-style minimization of extended, cost-sensitive, losses. The framework is applied to the design of specific algorithms, by introduction of cost-sensitive extensions of the exponential and binomial losses. Minimization of these losses leads to cost sensitive extensions of the popular AdaBoost, RealBoost, and LogitBoost algorithms. Experimental validation, on various UCI datasets and the computer vision problem of face detection, shows that the new algorithms substantially improve performance over what was achievable with previous cost-sensitive boosting approaches.Author

Calibrating AdaBoost for Asymmetric Learning

“…As for the other variants, they are all methods that modify the training algorithm. CSB0 and CSB1 [17] do not use confidence rated predictions and based on the results of comparative studies [9,10,15], the two variants are typically dominated by CSB2. Asymmetric-Adaboost [18] was excluded from said studies as being similar to CSB2.…”

“…AdaCost [2,17] is also outperformed by AdaC2 and CSB2 and so is AdaC3 [16]. CS-AdaBoost [9,10], despite being the only method other than AdaC2 with a solid theoretical basis, has been characterized as 'time-consuming and imprecise' [19], as it lacks a closed form solution for α t and the optimization of its parameters is therefore computationally intensive.…”

“…To our knowledge, the only previous attempt at directly comparing asymmetric AdaBoost variants to calibrated AdaBoost was by Masnadi-Shirazi and Vasconselos [10]. The comparison was performed on imbalanced data, it included AdaC2 and CSB2 and the performance of calibrated AdaBoost was found to be slightly inferior to theirs.…”

“…However it is often regarded as skew-insensitive [15,17], meaning it is unable to handle asymmetric tasks. There exist many skew-sensitive AdaBoost variants, including AdaCost [2,17], CSB0, CSB1, CSB2 [17], Asymmetric-Adaboost [18], RareBoost [6], AdaC1, AdaC2, AdaC3 [16], CS-AdaBoost [9,10]. However, most of them are heuristic and as a result they lack the theoretical guarantees of the original AdaBoost [7].…”

Learning from streaming data with concept drift and imbalance: an overview

“…Bagging [10] and boosting [30] are two popular methods for building ensembles of classifiers with a rich history of extensions [17,31,39,61,74,78]. In this section we outline various approaches which have been taken to make bagging and boosting methods overcome concept drift.…”