Rediscovery rate estimation for assessing the validation of significant findings in high-throughput studies

Ganna, Andrea; Lee, Donghwan; Pawitan, Yudi

doi:10.1093/bib/bbu033

Cited by 26 publications

(37 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Define

R_{t} = \sum_{i = 1}^{N} 1 ((), | T_{ti} | > c)

and

R_{v} = \sum_{i = 1}^{N} 1 ((), | T_{ti} | > c, | T_{vi} | > k)

, where c is a critical value in the training sample and k in the validation sample. Ganna et al , defined RDR as the expected proportion of statistically significant results in the training sample that can be validated, that is,

RDR = E (R_{v} / R_{t} ∣ R_{t} > 0) .

We now find a nonparametric representation for RDR, which shows its relationship to the FDR. Because FDR is a well‐known concept, this relationship is useful for understanding the RDR intuitively.…”

Section: A Nonparametric Approach To Estimating Rdrmentioning

confidence: 81%

“…Ganna et al , assumed T t i and T v i to be the t ‐statistics, but in this paper, we do not make such an assumption. Let n t 0 and n t 1 be the sample sizes of the control group and case group in the training sample; and similarly n v 0 and n v 1 in the validation sample.…”

Section: A Nonparametric Approach To Estimating Rdrmentioning

confidence: 99%

“…Recently, Ganna et al , used a parametric approach based on a t ‐mixture model to estimate α v and 1 − β v . For the distribution of T t i ,

f_{t} (t) = \sum_{j = 0}^{m} π_{j} g_{j, t} (t),

where

g_{j, t} (t) = f (t; \sqrt{n_{t 0} n_{t 1} / ((), n_{t 0} + n_{t 1})} δ_{j} / σ, n_{t} - 2)

and π j is the mixture probability for the j th group.…”

Section: Estimationmentioning

confidence: 99%

“…Similarly, for T v i ,

f_{v} (t) = \sum_{j = 0}^{m} π_{j} g_{j, v} (t),

where

g_{j, v} (t) = f (t; \sqrt{n_{v 0} n_{v 1} / (n_{v 0} + n_{v 1})} δ_{j} / σ, n_{v} - 2)

. From these, we have

α_{v} = \int_{| t | > k} g_{0, v} (t) dt

and

1 - β_{v} = \frac{1}{1 - π_{0}} \sum_{j = 1}^{m} π_{j} \int_{| t | > k} g_{j, v} (t) dt .

The parametric approach of Ganna et al , works well if the distribution assumption is appropriate, and the number of mixture components m is selected correctly. But, in practice, it is often difficult to guarantee the correct model specification, and this mixture model becomes problematic when π 0 is near 1.…”

Section: Estimationmentioning

confidence: 99%

See 3 more Smart Citations

Nonparametric estimation of the rediscovery rate

Lee

Ganna

Pawitan

et al. 2016

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

Validation studies have been used to increase the reliability of the statistical conclusions for scientific discoveries; such studies improve the reproducibility of the findings and reduce the possibility of false positives. Here, one of the important roles of statistics is to quantify reproducibility rigorously. Two concepts were recently defined for this purpose: (i) rediscovery rate (RDR), which is the expected proportion of statistically significant findings in a study that can be replicated in the validation study and (ii) false discovery rate in the validation study (vFDR). In this paper, we aim to develop a nonparametric approach to estimate the RDR and vFDR and show an explicit link between the RDR and the FDR. Among other things, the link explains why reproducing statistically significant results even with low FDR level may be difficult. Two metabolomics datasets are considered to illustrate the application of the RDR and vFDR concepts in high-throughput data analysis. Copyright © 2016 John Wiley & Sons, Ltd.

show abstract

“…Define

R_{t} = \sum_{i = 1}^{N} 1 ((), | T_{ti} | > c)

and

R_{v} = \sum_{i = 1}^{N} 1 ((), | T_{ti} | > c, | T_{vi} | > k)

RDR = E (R_{v} / R_{t} ∣ R_{t} > 0) .

Section: A Nonparametric Approach To Estimating Rdrmentioning

confidence: 81%

Section: A Nonparametric Approach To Estimating Rdrmentioning

confidence: 99%

“…Recently, Ganna et al , used a parametric approach based on a t ‐mixture model to estimate α v and 1 − β v . For the distribution of T t i ,

f_{t} (t) = \sum_{j = 0}^{m} π_{j} g_{j, t} (t),

where

g_{j, t} (t) = f (t; \sqrt{n_{t 0} n_{t 1} / ((), n_{t 0} + n_{t 1})} δ_{j} / σ, n_{t} - 2)

and π j is the mixture probability for the j th group.…”

Section: Estimationmentioning

confidence: 99%

“…Similarly, for T v i ,

f_{v} (t) = \sum_{j = 0}^{m} π_{j} g_{j, v} (t),

where

g_{j, v} (t) = f (t; \sqrt{n_{v 0} n_{v 1} / (n_{v 0} + n_{v 1})} δ_{j} / σ, n_{v} - 2)

. From these, we have

α_{v} = \int_{| t | > k} g_{0, v} (t) dt

and

1 - β_{v} = \frac{1}{1 - π_{0}} \sum_{j = 1}^{m} π_{j} \int_{| t | > k} g_{j, v} (t) dt .

Section: Estimationmentioning

confidence: 99%

See 2 more Smart Citations

Nonparametric estimation of the rediscovery rate

Lee

Ganna

Pawitan

et al. 2016

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

show abstract

“…Analyses of single vitamin and nutrient supplement use were adjusted for multivitamin use. To account for multiple comparisons while also allowing for biomarker discovery and hypothesis generation, a false discovery rate (FDR) , 0.1 was applied (Benjamini-Hochberg procedure) (36,37). Diet-metabolite correlations first were ranked from smallest to highest P value.…”

Section: Statisticsmentioning

confidence: 99%

Comparing metabolite profiles of habitual diet in serum and urine

Playdon

Sampson

Cross

et al. 2016

The American Journal of Clinical Nutrition

148

124

View full text Add to dashboard Cite

Background: Diet plays an important role in chronic disease etiology, but some diet-disease associations remain inconclusive because of methodologic limitations in dietary assessment. Metabolomics is a novel method for identifying objective dietary biomarkers, although it is unclear what dietary information is captured from metabolites found in serum compared with urine. Objective: We compared metabolite profiles of habitual diet measured from serum with those measured from urine. Design: We first estimated correlations between consumption of 56 foods, beverages, and supplements assessed by a food-frequency questionnaire, with 676 serum and 848 urine metabolites identified by untargeted liquid chromatography mass spectrometry, ultra-high performance liquid chromatography tandem mass spectrometry, and gas chromatography mass spectrometry in a colon adenoma case-control study (n = 125 cases and 128 controls) while adjusting for age, sex, smoking, fasting, case-control status, body mass index, physical activity, education, and caloric intake. We controlled for multiple comparisons with the use of a false discovery rate of ,0.1. Next, we created serum and urine multiple-metabolite models to predict food intake with the use of 10-fold crossvalidation least absolute shrinkage and selection operator regression for 80% of the data; predicted values were created in the remaining 20%. Finally, we compared predicted values with estimates obtained from self-reported intake for metabolites measured in serum and urine. Results: We identified metabolites associated with 46 of 56 dietary items; 417 urine and 105 serum metabolites were correlated with $1 food, beverage, or supplement. More metabolites in urine (n = 154) than in serum (n = 39) were associated uniquely with one food. We found previously unreported metabolite associations with leafy green vegetables, sugar-sweetened beverages, citrus, added sugar, red meat, shellfish, desserts, and wine. Prediction of dietary intake from multiple-metabolite profiles was similar between biofluids. Conclusions: Candidate metabolite biomarkers of habitual diet are identifiable in both serum and urine. Urine samples offer a valid alternative or complement to serum for metabolite biomarkers of diet in large-scale clinical or epidemiologic studies.Am J Clin Nutr 2016;104:776-89.

show abstract

The metabolites urobilin and sphingomyelin (30:1) are associated with incident heart failure in the general population

et al. 2019

Self Cite

View full text Add to dashboard Cite

Aims We aimed to investigate whether metabolomic profiling of blood can lead to novel insights into heart failure pathogenesis or improved risk prediction. Methods and results Mass spectrometry‐based metabolomic profiling was performed in plasma or serum samples from three community‐based cohorts without heart failure at baseline (total n = 3924; 341 incident heart failure events; median follow‐up ranging from 4.6 to 13.9 years). Cox proportional hazard models were applied to assess the association of each of the 206 identified metabolites with incident heart failure in the discovery cohorts Prospective Investigation of the Vasculature in Uppsala Seniors (PIVUS) (n = 920) and Uppsala Longitudinal Study of Adult Men (ULSAM) (n = 1121). Replication was undertaken in the independent cohort TwinGene (n = 1797). We also assessed whether metabolites could improve the prediction of heart failure beyond established risk factors (age, sex, body mass index, low‐density and high‐density lipoprotein cholesterol, triglycerides, lipid medication, diabetes, systolic and diastolic blood pressure, blood pressure medication, glomerular filtration rate, smoking status, and myocardial infarction prior to or during follow‐up). Higher circulating urobilin and lower sphingomyelin (30:1) were associated with incident heart failure in age‐adjusted and sex‐adjusted models in the discovery and replication sample. The hazard ratio for urobilin in the replication cohort was estimated to 1.29 per standard deviation unit, 95% confidence interval (CI 1.03–1.63), and for sphingomyelin (30:1) to 0.72 (95% CI 0.58–0.89). Results remained similar after further adjustment for established heart failure risk factors in meta‐analyses of all three cohorts. Urobilin concentrations were inversely associated with left ventricular ejection fraction at baseline in the PIVUS cohort (β = −0.70, 95% CI −1.03 to −0.38). No major improvement in risk prediction was observed when adding the top 2 metabolites (C‐index 0.787, 95% CI 0.752–0.823) or nine Lasso‐selected metabolites (0.790, 95% CI 0.754–0.826) to a modified Atherosclerosis Risk in Communities heart failure risk score model (0.780, 95% CI 0.745–0.816). Conclusions Our metabolomic profiling of three community‐based cohorts study identified associations of circulating levels of the haem breakdown product urobilin, and sphingomyelin (30:1), a cell membrane component involved in signal transduction and apoptosis, with incident heart failure.

show abstract

Rediscovery rate estimation for assessing the validation of significant findings in high-throughput studies

Cited by 26 publications

References 24 publications

Nonparametric estimation of the rediscovery rate

Nonparametric estimation of the rediscovery rate

Comparing metabolite profiles of habitual diet in serum and urine

The metabolites urobilin and sphingomyelin (30:1) are associated with incident heart failure in the general population

Contact Info

Product

Resources

About