k-Fold Cross-Validation Can Significantly Over-Estimate True Classification Accuracy in Common EEG-Based Passive BCI Experimental Designs: An Empirical Investigation

Abstract

In passive BCI studies, a common approach is to collect data from mental states of interest during relatively long trials and divide these trials into shorter “epochs” to serve as individual samples in classification. While it is known that using k-fold cross-validation (CV) in this scenario can result in unreliable estimates of mental state separability (due to autocorrelation in the samples derived from the same trial), k-fold CV is still commonly used and reported in passive BCI studies. What is not known is the extent to which k-fold CV misrepresents true mental state separability. This makes it difficult to interpret the results of studies that use it. Furthermore, if the seriousness of the problem were clearly known, perhaps more researchers would be aware that they should avoid it. In this work, a novel experiment explored how the degree of correlation among samples within a class affects EEG-based mental state classification accuracy estimated by k-fold CV. Results were compared to a ground-truth (GT) accuracy and to “block-wise” CV, an alternative to k-fold which is purported to alleviate the autocorrelation issues. Factors such as the degree of true class separability and the feature set and classifier used were also explored. The results show that, under some conditions, k-fold CV inflated the GT classification accuracy by up to 25%, but block-wise CV underestimated the GT accuracy by as much as 11%. It is our recommendation that the number of samples derived from the same trial should be reduced whenever possible in single-subject analysis, and that both the k-fold and block-wise CV results are reported.

Fusion of Deep Features from 2D-DOST of fNIRS Signals for Subject-Independent Classification of Motor Execution Tasks

“…Also, the results of fve-fold cross-validation protocol are given Table 13. As observed, this protocol outperforms the cross-subject cross-validation, while due to the following reasons, the cross-subject protocol is most popular than the k-fold one in BCI applications [32][33][34].…”

CT-based delta-radiomics nomogram to predict pathological complete response after neoadjuvant chemoradiotherapy in esophageal squamous cell carcinoma patients

“…However, in machine learning models based on radiomics, k-fold cross-validation risks data leakage and introduces errors. Also, inappropriate k values, imbalanced training and validation set division, and unshuffled data order affect model prediction [ 59 , 60 ]. This study adopted random grouping and k-fold cross-validation for parameter selection during model building.…”

Advanced Machine Learning Techniques for Predictive Modeling of Property Prices

“…For model evaluation, we employ k-fold cross-validation, indicated by the cv = 5 parameter in the RandomizedSearchCV function. By specifying cv = 5, we perform 5-fold cross-validation, iteratively splitting the training data into five equal-sized folds for training and validation [37]. Utilizing multiple folds enhances performance estimates' robustness and reduces variability compared to a single train-test split.…”