Meta-analysis using effect size distributions of only statistically significant studies.

Abstract

Publication bias threatens the validity of meta-analytic results and leads to overestimation of the effect size in traditional meta-analysis. This particularly applies to meta-analyses that feature small studies, which are ubiquitous in psychology. Here we develop a new method for meta-analysis that deals with publication bias. This method, p-uniform, enables (a) testing of publication bias, (b) effect size estimation, and (c) testing of the null-hypothesis of no effect. No current method for meta-analysis possesses all 3 qualities. Application of p-uniform is straightforward because no additional data on missing studies are needed and no sophisticated assumptions or choices need to be made before applying it. Simulations show that p-uniform generally outperforms the trim-and-fill method and the test of excess significance (TES; Ioannidis & Trikalinos, 2007b) if publication bias exists and population effect size is homogenous or heterogeneity is slight. For illustration, p-uniform and other publication bias analyses are applied to the meta-analysis of McCall and Carriger (1993) examining the association between infants' habituation to a stimulus and their later cognitive ability (IQ). We conclude that p-uniform is a valuable technique for examining publication bias and estimating population effects in fixed-effect meta-analyses, and as sensitivity analysis to draw inferences about publication bias.

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

“…If a result is inconsistent it is often impossible (in the absence of raw data) to determine whether the test statistic, the degrees of freedom, or the p-value were incorrectly reported. If the test statistic is incorrect and it is used to calculate the effect size for a meta-analysis, this effect size will be incorrect as well, which could affect the outcome of the meta-analysis ; in fact, the misreporting of all kinds of statistics is a problem for meta-analyses; Gotzsche, Hrobjartsson, Maric, & Tendal, 2007;Levine & Hullett, 2002 (Simonsohn, Nelson, & Simmons, 2014) and p-uniform (van Assen, van Aert, & Wicherts, 2015). Moreover, Wicherts et al (2011) reported that a higher prevalence of reporting errors was associated with a failure to share data upon request.…”

Too Good to be False: Nonsignificant Results Revisited

“…Hence, we expect little p-hacking and substantial evidence of false negatives in reported gender effects in psychology. We apply the Fisher test to significant and nonsignificant gender results to test for evidential value (van Assen, van Aert, & Wicherts, 2015;Simonsohn, Nelson, & Simmons, 2014). More precisely, we investigate whether evidential value depends on whether or not the result is statistically significant, and whether or not the results were in line with expectations expressed in the paper.…”

“…Statistically nonsignificant results were transformed with Equation 1; statistically significant p-values were divided by alpha (.05; van Assen, van Aert, & Wicherts, 2015;Simonsohn, Nelson, & Simmons, 2014). …”

“…Whereas Fisher used his method to test the null-hypothesis of an underlying true zero effect using several studies' p-values, the method has recently been extended to yield unbiased effect estimates using only statistically significant p-values. The principle of uniformly distributed p-values given the true effect size on which the Fisher method is based, also underlies newly developed methods of meta-analysis that adjust for publication bias, such as p-uniform (van Assen, van Aert, & Wicherts, 2015) and p-curve (Simonsohn, Nelson, & Simmons, 2014). Extensions of these methods to include nonsignificant as well as significant p-values and to estimate heterogeneity are still under construction.…”

“…This overemphasis is substantiated by the finding that more than 90% of results in the psychological literature are statistically significant (Open Science Collaboration, 2015;Sterling, Rosenbaum, & Weinkam, 1995;Sterling, 1959) despite low statistical power due to small sample sizes (Cohen, 1962;Sedlmeier, & Gigerenzer, 1989;Marszalek, Barber, Kohlhart, & Holmes, 2011;Bakker, van Dijk, & Wicherts, 2012). Consequently, publications have become biased by overrepresenting statistically significant results (Greenwald, 1975), which generally results in effect size overestimation in both individual studies (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2015) and meta-analyses (van Assen, van Aert, & Wicherts, 2015;Lane, & Dunlap, 1978;Rothstein, Sutton, & Borenstein, 2005;Borenstein, Hedges, Higgins, & Rothstein, 2009). The overemphasis on statistically significant effects has been accompanied by questionable research practices (QRPs; John, Loewenstein, & Prelec, 2012) such as erroneously rounding p-values towards significance, which for example occurred for 13.8% of all p-values reported as "p = .05" in articles from eight major psychology journals in the period 1985-2013 (Hartgerink, van Aert, Nuijten, Wicherts, & van Assen, 2016).…”