A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning

Abstract

Nowadays, the solution to many practical problems relies on machine learning tools. However, compiling the appropriate training data set for real-world classification problems is challenging because collecting the right amount of data for each class is often difficult or even impossible. In such cases, we can easily face the problem of imbalanced learning. There are many methods in the literature for solving the imbalanced learning problem, so it has become a serious question how to compare the performance of the imbalanced learning methods. Inadequate validation techniques can provide misleading results (e.g., due to data shift), which leads to the development of methods designed for imbalanced data sets, such as stratified cross-validation (SCV) and distribution optimally balanced SCV (DOB-SCV). Previous studies have shown that higher classification performance scores (AUC) can be achieved on imbalanced data sets using DOB-SCV instead of SCV. We investigated the effect of the oversamplers on this difference. The study was conducted on 420 data sets, involving several sampling methods and the DTree, kNN, SVM, and MLP classifiers. We point out that DOB-SCV often provides a little higher F1 and AUC values for classification combined with sampling. However, the results also prove that the selection of the sampler–classifier pair is more important for the classification performance than the choice between the DOB-SCV and the SCV techniques.

Time‐domain heart rate variability features for automatic congestive heart failure prediction

“…However, to address the challenge associated with severely imbalanced datasets that could cause some folds not containing elements from all classes, the stratified cross‐validation method is used. This method preserves the percentage of samples from majority and minority classes by splitting the dataset on k folds 33 . The stratified five‐fold cross‐validation ensures that the proportion of instances (healthy and heart failure recordings) is preserved in each partition.…”

Predicting Anorexia Nervosa and Autistic Characteristics in Individuals with Anorexia Nervosa from Resting State Cyclic Connectivity

“…Several classification model algorithms were built to discover neurophenotypes underlying AN. Models were scored by accuracy and area under the receiver operating curve (AUC ROC) using stratified cross-validation, which randomly samples from the minority and majority classes to the original distribution, so each fold has the same distribution as the initial distribution, achieving more robust validation (Szeghalmy & Fazekas, 2023). As our classes for group classification where imbalanced (HC = 27, AN = 65), which is known to cause bias (Teh, Armitage, Tesfaye, Selvarajah & Wilkinson 2020), the oversampling method assembled synthetic minority oversampling technique was used.…”

Multimodality radiomics prediction of radiotherapy-induced the early proctitis and cystitis in rectal cancer patients: a machine learning study

“…Subsequently, eight different classification methods, detailed in table 1, were utilized to construct prediction models. The stratified 5-fold cross-validation method [32,33] was adopted, where all features were incorporated into the feature selection algorithms. A set of features was chosen for each fold.…”