Optimal Strategies for Sequential Validation of Significant Features from High-Dimensional Genomic Data

Abstract

High-dimensional genomic studies play a key role in identifying critical features that are significantly associated with a phenotypic outcome. The two most important examples are the detection of (1) differentially expressed genes from genome-wide gene expression studies and (2) single-nucleotide polymorphisms (SNPs) from genome-wide association studies. Such experiments are often associated with high noise levels, and the validity of statistical conclusions suffers from low sample size compared to large number of features. The corresponding multiple testing problem calls for the identification of optimal strategies for controlling the numbers of false discoveries and false nondiscoveries. In addition, a frequent validation problem is that features identified as important in one study are often less so in another study. Adjustment for multiple testing in both studies separately increases the risk of missing the crucial features even further. These problems can be addressed by sequential validation strategies, where only significant features identified in one study enter as candidates in the next study. The quality associated with different studies, for example, in terms of noise levels, may vary considerably. By performing simulation studies it is possible to demonstrate that the optimal order for this stepwise procedure is to sort experimental studies according to their quality in descending order. The impact of the method for multiple testing adjustment (Bonferroni-Holm, FDR) was also analyzed. Finally, the sequential validation strategy was applied to three large breast cancer studies with gene expression measurements, confirming the crucial impact of the order of the validation steps in a real-world application.

Elevated mRNA Levels of AURKA, CDC20 and TPX2 are associated with poor prognosis of smoking related lung adenocarcinoma using bioinformatics analysis

“…It suggested that elevated expression of these three genes were associated with poor OS among the smoking related lung adenocarcinoma patients. However, there is a puzzle that the identified genes in training cohorts could not easily be validated in external cohorts 41. One reason might be the effects of genes have broad confidence intervals so that it is difficult to identified using a single validation database.…”

Epsin Family Member 3 and Ribosome-Related Genes Are Associated with Late Metastasis in Estrogen Receptor-Positive Breast Cancer and Long-Term Survival in Non-Small Cell Lung Cancer Using a Genome-Wide Identification and Validation Strategy

“…One strategy to address the problem of multiple testing is sequential validation, where significant genes identified in a discovery set enter as candidates in a validation set. We previously recommended an optimized order for such a stepwise procedure, where the datasets with the largest sample size (and the lowest measurement variance) are used for discovery steps and the datasets with the smallest sample size for validation steps [32]. Based on this approach, the Rotterdam and Transbig cohorts were here used for gene discovery and the Mainz cohort for validation.…”

Biomarker Discovery in Non–Small Cell Lung Cancer: Integrating Gene Expression Profiling, Meta-analysis, and Tissue Microarray Validation

“…As a consequence, there is high confidence in the relevance of the final candidates. However, it has been shown that this approach yields many false negative results (19). Instead, we applied in this study a combined sequential and meta-analysis approach, in which candidate biomarkers with prognostic relevance in the primary cohort were further evaluated in a meta-analysis of several publicly available datasets.…”

“…Surprisingly, there is virtually no overlap between hitherto published gene signatures. An explanation might be that effects of single genes with broad confidence intervals are difficult to confirm using a sequential validation strategy, that is, when genes identified as significant in one study are tested for significance in separate subsequent studies of comparably small sample size (19).…”