An empirical assessment of validation practices for molecular classifiers

Abstract

Proposed molecular classifiers may be overfit to idiosyncrasies of noisy genomic and proteomic data. Cross-validation methods are often used to obtain estimates of classification accuracy, but both simulations and case studies suggest that, when inappropriate methods are used, bias may ensue. Bias can be bypassed and generalizability can be tested by external (independent) validation. We evaluated 35 studies that have reported on external validation of a molecular classifier. We extracted information on study design and methodological features, and compared the performance of molecular classifiers in internal cross-validation versus external validation for 28 studies where both had been performed. We demonstrate that the majority of studies pursued cross-validation practices that are likely to overestimate classifier performance. Most studies were markedly underpowered to detect a 20% decrease in sensitivity or specificity between internal cross-validation and external validation [median power was 36% (IQR, 21-61%) and 29% (IQR, 15-65%), respectively]. The median reported classification performance for sensitivity and specificity was 94% and 98%, respectively, in cross-validation and 88% and 81% for independent validation. The relative diagnostic odds ratio was 3.26 (95% CI 2.04-5.21) for cross-validation versus independent validation. Finally, we reviewed all studies (n = 758) which cited those in our study sample, and identified only one instance of additional subsequent independent validation of these classifiers. In conclusion, these results document that many cross-validation practices employed in the literature are potentially biased and genuine progress in this field will require adoption of routine external validation of molecular classifiers, preferably in much larger studies than in current practice.

COPD subtypes identified by network-based clustering of blood gene expression

“…However, in that study, class discovery was repeated in the replication sample as opposed to the approach used in this analysis where all key model parameters were learned in the training data and directly transferred to the replication set. The latter approach provides a more stringent assessment of generalizability [4,16]. …”

“…Signatures for several diseases are in clinical use, but many gene expression signatures are poorly reproducible and suffer from sampling dependent instability [1,2]. While inappropriate analysis methods account for some of the poor reproducibility of published signatures [3,4], another potential cause is the presence of unrecognized biologic variability, such as occult molecular disease subtypes. Interestingly, one perspective on gene expression data is that it contains too much information, i.e.…”

A Five-Gene Hedgehog Signature Developed as a Patient Preselection Tool for Hedgehog Inhibitor Therapy in Medulloblastoma

“…Other studies identifying and analyzing gene signatures have shown impressive results; however, these studies often lacked rigorous data analysis and validation (26). Conversely, the gene signature, based on data from previous gene expression profiling studies, was independently selected and validated in separate settings.…”

Analytical strategies for studying stem cell metabolism

“…This procedure is repeated many times until each sample has been in the test set exactly once. The accuracy of the model on the test samples gives an estimate of the predictive power and the robustness of the model to perturbations of the data (Castaldi et al, 2011; Ioannidis and Khoury, 2011). …”