P‐hacking in meta‐analyses: A formalization and new meta‐analytic methods

Abstract

As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta‐analyses can also be compromised by selection within studies (SWS), in which investigators “p‐hack” results within their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta‐analytic mean by fitting “right‐truncated meta‐analysis” (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta‐analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta‐analyses to joint SAS and SWS.

Assessing robustness to worst case publication bias using a simple subset meta-analysis

“…Conducting MAN involves simply meta-analyzing only the non-affirmative studies and excluding all affirmative studies 1822. No special statistical methods are required—a standard meta-analysis is simply conducted of only the non-affirmative studies.…”

“…In particular, if the studies are P hacked in a manner that favors affirmative results, then the MAN estimate is still conservative (ie, attenuated towards the null). In contrast, standard methods for publication bias could be severely biased either towards or away from the null 18. Thus, MAN can provide a more holistic assessment of robustness to multiple forms of selective reporting than do standard methods for publication bias.…”

“…Thus, MAN can provide a more holistic assessment of robustness to multiple forms of selective reporting than do standard methods for publication bias. For example, a large simulation study assessed the performance of MAN in over 80 scenarios exhibiting numerous forms of selective reporting (comprising both publication bias and P-hacking), including many that were more complicated than the simple model assumed by MAN 18. These simulations found that the performance of MAN was robust to numerous plausible departures from its assumptions, whereas comparison methods (two-step selection models, PET-PEESE, and RoBMA) often had substantially compromised performance in scenarios with P-hacking 18.…”

“…These hybrid methods can provide some additional flexibility compared with funnel plot methods or selection models alone, but they also share many of their limitations. For example, RoBMA can perform worse than an uncorrected meta-analysis in the presence of P-hacking in addition to publication bias 18. The relative performance of funnel plot methods, selection models, and hybrid methods is further discussed below.…”

“…In fact, such P-hacking is quite common, even according to researchers’ self-admissions 21. When there is both traditional publication bias and P-hacking, standard methods for publication bias can be severely biased in either direction 18…”