Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation

Abstract

BackgroundWith the large amount of biological data that is currently publicly available, many investigators combine multiple data sets to increase the sample size and potentially also the power of their analyses. However, technical differences (“batch effects”) as well as differences in sample composition between the data sets may significantly affect the ability to draw generalizable conclusions from such studies.FocusThe current study focuses on the construction of classifiers, and the use of cross-validation to estimate their performance. In particular, we investigate the impact of batch effects and differences in sample composition between batches on the accuracy of the classification performance estimate obtained via cross-validation. The focus on estimation bias is a main difference compared to previous studies, which have mostly focused on the predictive performance and how it relates to the presence of batch effects.DataWe work on simulated data sets. To have realistic intensity distributions, we use real gene expression data as the basis for our simulation. Random samples from this expression matrix are selected and assigned to group 1 (e.g., ‘control’) or group 2 (e.g., ‘treated’). We introduce batch effects and select some features to be differentially expressed between the two groups. We consider several scenarios for our study, most importantly different levels of confounding between groups and batch effects.MethodsWe focus on well-known classifiers: logistic regression, Support Vector Machines (SVM), k-nearest neighbors (kNN) and Random Forests (RF). Feature selection is performed with the Wilcoxon test or the lasso. Parameter tuning and feature selection, as well as the estimation of the prediction performance of each classifier, is performed within a nested cross-validation scheme. The estimated classification performance is then compared to what is obtained when applying the classifier to independent data.

FERAL: network-based classifier with application to breast cancer outcome prediction

“…A fundamental limitation of breast cancer outcome prediction is that it has proved very difficult to obtain a robust classifier performance across different datasets. It was found that, despite properly cross-validated classifier training, prediction performance decreases dramatically when a classifier trained on one dataset is applied to another one ( Lazar et al , 2013 ; Soneson et al , 2014 ). Moreover, the prognostic gene signatures identified using these classifiers have poor concordance across different studies ( Ein-Dor et al , 2005 ; van Vliet et al , 2008 ).…”

Applications of Bayesian network models in predicting types of hematological malignancies

“…However, these datasets do not include MDS samples; therefore, they could not be used for the validation of our binary classifier. Combining these datasets with the MDS samples from other datasets could be problematic because of possible batch effects 71 . Also, the expression profile of 159 MDS cases in the GSE58831 dataset 72 could not be directly compared to the MDS samples analyzed in this study because of the difference in the tissues.…”

De novo assembly, annotation, and characterization of the whole brain transcriptome of male and female Syrian hamsters

“…This sample size supplies sufficient power for downstream analyses while minimizing the total number of animals sacrificed4445. We used Trizol (Life Technologies, Grand Island, NY) for extractions, following a modified version of the manufacturer’s protocol.…”