Supervised classification in the presence of misclassified training data: a Monte Carlo simulation study in the three group case

Abstract

Statistical classification of phenomena into observed groups is very common in the social and behavioral sciences. Statistical classification methods, however, are affected by the characteristics of the data under study. Statistical classification can be further complicated by initial misclassification of the observed groups. The purpose of this study is to investigate the impact of initial training data misclassification on several statistical classification and data mining techniques. Misclassification conditions in the three group case will be simulated and results will be presented in terms of overall as well as subgroup classification accuracy. Results show decreased classification accuracy as sample size, group separation and group size ratio decrease and as misclassification percentage increases with random forests demonstrating the highest accuracy across conditions.

Successional Stages of Santa Catarina Atlantic Subtropical Evergreen Rainforest: A Classification Method Proposal

“…Analyzing groups with different numerical ratios (such as 10:90 or 25:75; Table 2) can significantly impair the discriminant potential of LDA, with higher probability of samples being reclassified within the larger group, and less likely to be classified in the smaller group (Holden et al, 2011;Bolin and Finch, 2014). Thus, we excluded four stands classified in the early stage by CONAMA and Scenario A schemes of the performance analysis and focused our analysis on the discrimination between the intermediate and late stages.…”

Prediction With Mixed Effects Models: A Monte Carlo Simulation Study

“…While no unitary sample size conditions were employed, these effects illustrate a pattern somewhat different from single-level classifiers. That is, previous studies comparing single-level classifiers (e.g., Bolin & Finch, 2014; Holden et al, 2011; Lei & Koehly, 2003) have found that as the sample size increases, estimates tend to become more stable but do not necessarily increase appreciably in accuracy. However, in the multilevel context, it is evident that a higher number of clusters and cases per cluster is associated with noticeable increases in accuracy.…”

“…Sela and Simonoff (2012) utilized varied L1 and L2 sample sizes with the former featuring values from 50 to 2,000 and the latter featuring values from 10 to 100; Ngufor et al (2019) also incorporated varied numbers of clusters and time points from 10 to 60 clusters and 1 to 4 time points (within-cluster sample size). Additionally, Bolin and Finch (2014), Holden et al (2011), and Lei and Koehly (2003) each considered varied parameters in single-level classification situations including varied group size ratios and group separation, among other factors, in simulated data. The salient effects of the ICC must also be considered in multilevel contexts, as this is precisely the factor that necessitates use of mixed effects models (with LeBreton & Senter’s [2008] recommendation being any ICC > 0.05 necessitating multilevel models; Raudenbush & Bryk 2002).…”

“…Additionally, the impact of the number of clusters has yet to be fully determined in the context of classification analyses. Studies considering single-level classifiers have found that increased sample sizes tend to yield more stable results across iterations within simulation studies (Bolin & Finch, 2014; Pohar et al, 2004), but no analogous results have been found for multilevel classifiers. Also uniquely pertaining to multilevel contexts is the ICC and between-cluster correlations.…”

“…Unbalanced outcome group sizes—such as those found in situations of predicting student dropout (Morris et al, 2005; U.S. Department of Education, National Center for Education Statistics, 2018), severe mental illness (National Institute of Mental Health, 2019), and suicide attempts (Hedegaard et al, 2018; Ribeiro et al, 2016)—as well as poorly separated groups, have the effect of reducing model prediction accuracy (Bolin & Finch, 2014; Holden et al, 2011; Lei & Koehly, 2003). Under the condition of unbalanced groups, classifiers tend to yield high overall accuracy and large-group recovery (LGR) rates but with a concomitant diminution of small-group recovery (SGR) rates (Kessler et al, 2003; Lei & Koehly, 2003; Mann et al, 2008; Morris et al, 2005).…”