Optimal Trees Selection for Classification via Out-of-Bag Assessment and Sub-Bagging

Abstract

The effect of training data size on machine learning methods has been well investigated over the past two decades. The predictive performance of tree based machine learning methods, in general, improves with a decreasing rate as the size of training data increases. We investigate this in optimal trees ensemble (OTE) where the method fails to learn from some of the training observations due to internal validation. Modified tree selection methods are thus proposed for OTE to cater for the loss of training observations in internal validation. In the first method, corresponding out-of-bag (OOB) observations are used in both individual and collective performance assessment for each tree. Trees are ranked based on their individual performance on the OOB observations. A certain number of top ranked trees is selected and starting from the most accurate tree, subsequent trees are added one by one and their impact is recorded by using the OOB observations left out from the bootstrap sample taken for the tree being added. A tree is selected if it improves predictive accuracy of the ensemble. In the second approach, trees are grown on random subsets, taken without replacement-known as sub-bagging, of the training data instead of bootstrap samples (taken with replacement). The remaining observations from each sample are used in both individual and collective assessments for each corresponding tree similar to the first method. Analysis on 21 benchmark datasets and simulations studies show improved performance of the modified methods in comparison to OTE and other state-of-the-art methods.

Prediction of Multidrug-Resistant Tuberculosis Using Machine Learning Algorithms in SWAT, Pakistan

“…ML techniques were used to classify MTBC smearnegative pulmonary-TB patients by Mello et al [13]. e span of the study was 3 years from April 1, 1995, to December 31,1998. A toal of 551 cases were considered for the analysis recording current TB symptoms and radiological information such as cavities above the lungs.…”

“…In the current research work, only demographic, medical, and psychological information is included, but for future work, chest X-rays and chest scans could also be considered for further insights. Additional feature selection methods [25][26][27][28][29] and classifiers [30,31] could also be used for further investigation. e same algorithms could also be extended for the prediction of binding of G-protein-coupled receptors (GPCRs) and ligands using machine learning algorithms [32] and biomedical image classification in a big data architecture [33].…”

SUBiNN: a stacked uni- and bivariate kNN sparse ensemble

“…Recently, there have been advances in developing interpretational tools for RF and other black box techniques (Sies and Van Mechelen 2020;Ribeiro et al 2016). Others have adapted ensemble methods with trees to increase interpretability (Meinshausen 2010) and searched for optimal tree ensembles (Khan et al 2020(Khan et al , 2021. We note that both in our simulation experiments as well as in the benchmark data set, there was at least one dataset or condition for which SUBiNN outperformed RF.…”

Robust proportional overlapping analysis for feature selection in binary classification within functional genomic experiments

“…The final set of genes, in that case, will be the combination of genes selected from all the clusters. Extending performance assessment of selected genes to other recent classification methods ( Khan et al, 2020a , Gul et al, 2018 ; Khanal et al, 2020 ; Khan et al, 2020b ) could further validate the proposed gene selection methods.…”