Meta-assessment of bias in science

Abstract

Numerous biases are believed to affect the scientific literature, but their actual prevalence across disciplines is unknown. To gain a comprehensive picture of the potential imprint of bias in science, we probed for the most commonly postulated bias-related patterns and risk factors, in a large random sample of meta-analyses taken from all disciplines. The magnitude of these biases varied widely across fields and was overall relatively small. However, we consistently observed a significant risk of small, early, and highly cited studies to overestimate effects and of studies not published in peer-reviewed journals to underestimate them. We also found at least partial confirmation of previous evidence suggesting that US studies and early studies might report more extreme effects, although these effects were smaller and more heterogeneously distributed across meta-analyses and disciplines. Authors publishing at high rates and receiving many citations were, overall, not at greater risk of bias. However, effect sizes were likely to be overestimated by early-career researchers, those working in small or long-distance collaborations, and those responsible for scientific misconduct, supporting hypotheses that connect bias to situational factors, lack of mutual control, and individual integrity. Some of these patterns and risk factors might have modestly increased in intensity over time, particularly in the social sciences. Our findings suggest that, besides one being routinely cautious that published small, highly-cited, and earlier studies may yield inflated results, the feasibility and costs of interventions to attenuate biases in the literature might need to be discussed on a discipline-specific and topic-specific basis.bias | misconduct | meta-analysis | integrity | meta-research

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

“…Previous research gave us sufficient reason to suspect that many effect sizes are overestimated (Button et al, 2013;Fanelli, 2010;Fanelli, Costas, & Ioannidis, 2017;Song et al, 2010). A big problem is that it is hard to determine for an individual study whether it contains an overestimated effect, and if so, how much it is overestimated.…”

“…And if we do suspect a study contains an overestimated effect, it is hard, if not impossible to determine if that is simply because of random sampling variation, or because of problems such as publication bias and QRPs. What we can do, however, is look for patterns of bias in meta-analyses (Fanelli et al, 2017;Rothstein, Sutton, & Borenstein, 2005;Song et al, 2010).…”

“…One way to investigate bias in science is by analyzing patterns in effect size estimates in meta-analyses (see, e.g., Fanelli et al, 2017;Fanelli & Ioannidis, 2013;Jennions & Moller, 2002). Here, we analyzed 2,439 effect sizes from 131 meta-analyses in intelligence research to estimate the average effect size, median power, and evidence for bias in this field.…”

“…Specifically, we looked at five types of bias that have been found to be problematic in other research fields (Fanelli et al, 2017): small study effect, US effect, decline effect, earlyextremes effect, and citation bias. We discuss these biases in detail below.…”

“…Many published findings cannot be replicated in novel samples (Klein et al, 2014;Open Science Collaboration, 2015), many meta-analyses highlight selective reporting of results depending on significance (Button et al, 2013;Fanelli et al, 2017;Niemeyer et al, 2012Niemeyer et al, , 2013, and the number of confirmed hypotheses in the literature is incompatible with the generally low statistical power of psychological studies Fanelli, 2010;Francis, 2014;Marszalek et al, 2011). It is argued that the main cause for this "replicability crisis" (Baker, 2016a) is a combination of publication bias and strategic use of flexibility in data analysis (Ioannidis, 2005;Munafò et al, 2017).…”