Discovering Findings That Replicate From a Primary Study of High Dimension to a Follow-Up Study

Bogomolov, Marina; Heller, Ruth

doi:10.1080/01621459.2013.829002

Cited by 30 publications

(62 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bogomolov and Heller considered the statistical problem of identifying whether findings replicate from one study of high dimension to another. To assess replicability from a validation study, Heller et al , defined the false discovery rate (FDR) for replicability analysis as the expected proportion of false replicability claims among all those called replicated.…”

Section: Introductionmentioning

confidence: 99%

Nonparametric estimation of the rediscovery rate

Lee

Ganna

Pawitan

et al. 2016

Statistics in Medicine

View full text Add to dashboard Cite

Validation studies have been used to increase the reliability of the statistical conclusions for scientific discoveries; such studies improve the reproducibility of the findings and reduce the possibility of false positives. Here, one of the important roles of statistics is to quantify reproducibility rigorously. Two concepts were recently defined for this purpose: (i) rediscovery rate (RDR), which is the expected proportion of statistically significant findings in a study that can be replicated in the validation study and (ii) false discovery rate in the validation study (vFDR). In this paper, we aim to develop a nonparametric approach to estimate the RDR and vFDR and show an explicit link between the RDR and the FDR. Among other things, the link explains why reproducing statistically significant results even with low FDR level may be difficult. Two metabolomics datasets are considered to illustrate the application of the RDR and vFDR concepts in high-throughput data analysis. Copyright © 2016 John Wiley & Sons, Ltd.

show abstract

Section: Introductionmentioning

confidence: 99%

Nonparametric estimation of the rediscovery rate

Lee

Ganna

Pawitan

et al. 2016

Statistics in Medicine

View full text Add to dashboard Cite

show abstract

“…Crossscreening performs poorly except when K is large and I is not extremely large, but it often wins decisively in sensitivity analyses with large K and moderate I/K. Cross-screening is related to, though distinct from, a concept of replicability developed by Bogomolov and Heller (2013); see §2.4 and §6 for detailed discussion. Cross-screening rejects hypothesis H k if either half-sample rejects H k , thereby strongly controlling the family-wise rate; however, it achieves Bogomolov-Heller replicability if both halves reject H k .…”

mentioning

confidence: 99%

Cross-Screening in Observational Studies That Test Many Hypotheses

Zhao

Small

Rosenbaum

2018

Journal of the American Statistical Association

View full text Add to dashboard Cite

We discuss observational studies that test many causal hypotheses, either hypotheses about many outcomes or many treatments. To be credible an observational study that tests many causal hypotheses must demonstrate that its conclusions are neither artifacts of multiple testing nor of small biases from nonrandom treatment assignment. In a sense that needs to be defined carefully, hidden within a sensitivity analysis for nonrandom assignment is an enormous correction for multiple testing: in the absence of bias, it is extremely improbable that multiple testing alone would create an association insensitive to moderate biases. We propose a new strategy called "cross-screening," different from but motivated by recent work of Bogomolov and Heller on replicability. Cross-screening splits the data in half at random, uses the first half to plan a study carried out on the second half, then uses the second half to plan a study carried out on the first half, and reports the more favorable conclusions of the two studies correcting using the Bonferroni inequality for having done two studies. If the two studies happen to concur, then they achieve Bogomolov-Heller replicability; however, importantly, replicability is not required for strong control of the family-wise error rate, and either study alone suffices for firm conclusions. In randomized studies with a few hypotheses, cross-split screening is not an attractive method when compared with conventional methods of multiplicity control, but it can become attractive when hundreds or thousands of hypotheses are subjected to sensitivity analyses in an observational study. We illustrate the technique by comparing 46 biomarkers in individuals who consume large quantities of fish versus little or no fish.

show abstract

“…However, research in the multiple-sequence setting has still focused on binary classification, typically on the problem of determining whether or not signals belong to class 3 of Table 1. This is of great interest because class 3 signals are more likely to constitute replicable scientific findings (Benjamini et al, 2009;Bogomolov and Heller, 2013;.…”

Section: Related Workmentioning

confidence: 99%

Signal Classification for the Integrative Analysis of Multiple Sequences of Large-Scale Multiple Tests

Xiang

Zhao

Cai

2019

Journal of the Royal Statistical Society Series B: Statistical Methodology

View full text Add to dashboard Cite

Summary The integrative analysis of multiple data sets is becoming increasingly important in many fields of research. When the same features are studied in several independent experiments, it can often be useful to analyse jointly the multiple sequences of multiple tests that result. It is frequently necessary to classify each feature into one of several categories, depending on the null and non‐null configuration of its corresponding test statistics. The paper studies this signal classification problem, motivated by a range of applications in large‐scale genomics. Two new types of misclassification rate are introduced, and two oracle procedures are developed to control each type while also achieving the largest expected number of correct classifications. Corresponding data‐driven procedures are also proposed, proved to be asymptotically valid and optimal under certain conditions and shown in numerical experiments to be nearly as powerful as the oracle procedures. In an application to psychiatric genetics, the procedures proposed are used to discover genetic variants that may affect both bipolar disorder and schizophrenia, as well as variants that may help to distinguish between these conditions.

show abstract

Discovering Findings That Replicate From a Primary Study of High Dimension to a Follow-Up Study

Cited by 30 publications

References 24 publications

Nonparametric estimation of the rediscovery rate

Nonparametric estimation of the rediscovery rate

Cross-Screening in Observational Studies That Test Many Hypotheses

Signal Classification for the Integrative Analysis of Multiple Sequences of Large-Scale Multiple Tests

Contact Info

Product

Resources

About