Generating highly accurate prediction hypotheses through collaborative ensemble learning

Abstract

Ensemble generation is a natural and convenient way of achieving better generalization performance of learning algorithms by gathering their predictive capabilities. Here, we nurture the idea of ensemble-based learning by combining bagging and boosting for the purpose of binary classification. Since the former improves stability through variance reduction, while the latter ameliorates overfitting, the outcome of a multi-model that combines both strives toward a comprehensive net-balancing of the bias-variance trade-off. To further improve this, we alter the bagged-boosting scheme by introducing collaboration between the multi-model’s constituent learners at various levels. This novel stability-guided classification scheme is delivered in two flavours: during or after the boosting process. Applied among a crowd of Gentle Boost ensembles, the ability of the two suggested algorithms to generalize is inspected by comparing them against Subbagging and Gentle Boost on various real-world datasets. In both cases, our models obtained a 40% generalization error decrease. But their true ability to capture details in data was revealed through their application for protein detection in texture analysis of gel electrophoresis images. They achieve improved performance of approximately 0.9773 AUROC when compared to the AUROC of 0.9574 obtained by an SVM based on recursive feature elimination.

Classification of the placement success in the undergraduate placement examination according to decision trees with bagging and boosting methods

“…The results obtained with the Bagging algorithm showed that in case of few outliers, these outliersin question are not very effective on the result. Similar results are drawing attentionwhen the data is scaled.Many studies reported similar results to these findings [32], [28], [33].Because the methods using Boosting algorithms show weak classification performance on adjusted variables according to variables that bears outliers and covariates.In contrast, the Bagging algorithm shows poor classification performance due to deviation from the sample [32].…”

Stability of decision trees and logistic regression

“…, and if LD (θ, λ) is ρ-Lipschitzian with respect to its first argument, we can then rewrite Equation (5)…”

“…As the previous paragraph states, hypothesis stability is the weakest of the three notions. Hypothesis stability has been derived for the k-nearest neighbors algorithm (k-NN) [3], for linear regression and regularized logistic regression in a technical report, for the AdaBoost algorithm [4], and for the Gentle Boost algorithm [5] as a constituent of a collaborative ensemble scheme [5,6]. L2 regularization with a penalty λ ∈ R + stabilizes learning algorithms because their objective functions then become λ-strongly convex, which leads to meeting the stability criterion.…”

Stacking and stability

“…An important virtue of stability-based bounds is their simplicity from a mathematical standpoint. Originally not as tight as the PAC-Bayes bounds [19], considered the tightest, they can be optimized or consolidated, i.e., be made significantly tighter in different ways, not the least of which is collaborative learning [25,26] that has attained significant generalization improvement.…”