Z-curve 2.0: Estimating Replication Rates and Discovery Rates

Abstract

Selection for statistical significance is a well-known factor that distorts the published literature and challenges the cumulative progress in science. Recent replication failures have fueled concerns that many published results are false-positives. Brunner and Schimmack (2020) developed z-curve, a method for estimating the expected replication rate (ERR) – the predicted success rate of exact replication studies based on the mean power after selection for significance. This article introduces an extension of this method, z-curve 2.0. The main extension is an estimate of the expected discovery rate (EDR) – the estimate of a proportion that the reported statistically significant results constitute from all conducted statistical tests. This information can be used to detect and quantify the amount of selection bias by comparing the EDR to the observed discovery rate (ODR; observed proportion of statistically significant results). In addition, we examined the performance of bootstrapped confidence intervals in simulation studies. Based on these results, we created robust confidence intervals with good coverage across a wide range of scenarios to provide information about the uncertainty in EDR and ERR estimates. We implemented the method in the zcurve R package (Bartoš & Schimmack, 2020).

Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field

“…However, reporting only effect sizes and their CIs, and full information about the pre-study power calculations, might not be enough. With the aim of facilitating cumulative scientific knowledge through meta-analysis [ 75 , 76 ], and the use of other statistical methods such as z -curve/ p -curve [ 44 , 84 ] or BUCSS to conduct power calculations adjusting for publication bias and uncertainty around parameter estimates [ 62 ], it has been suggested that besides sample size per condition, means, SDs and exact p -values, studies should also disclose F -ratio or t -statistics, the type of design, and the correlations between dependent observations for within-subjects designs [ 76 ], but it appears that this is rarely achieved. The compounding issues of poor reporting practices are easy to demonstrate with two examples; firstly, consider a within-subject design (i.e.…”

Prognostic and Predictive Biomarkers in Head and Neck Squamous Cell Carcinoma Treated with Radiotherapy—A Systematic Review

“…All data analysis was performed in R (v4.1.0) with packages: dplyr, ggplot2. Z curves were generated following the method described by Brunner, Schimmack and Bartoš [ 20 , 21 ], employing the R package zcurve. Literature search, data extraction, analysis and reporting were performed in accordance with the PRISMA and COSMOS-E statements [ 22 , 23 ].…”

Worse than expected: A z-curve reanalysis of motor cortex stimulation studies of embodied language comprehension

“…Based on these assumptions, z-curve takes a set of z-transformed p-values and models them as a mixture of truncated folded normal distributions (for details, see Bartoš & Schimmack, 2022) to calculate several indexes of interest.…”

“…Finally, it is also worth mentioning that TES tests for publication bias but cannot quantify the amount of bias (Bartoš & Schimmack, 2022).…”

“…Fortunately, a recently developed tool seems to be robust to these limitations: z-curve analysis (Bartoš & Schimmack, 2022;Brunner & Schimmack, 2020). Broadly, z-curve takes both significant and non-significant p-values as input (unlike p-curve, which only evaluates significant values), recomputes them into z-scores, and models them as a mixture of truncated folded normal distributions.…”