Detecting p-hacking

Abstract

We analyze what can be learned from tests for p-hacking based on distributions of t-statistics and p-values across multiple studies. We analytically characterize restrictions on these distributions that conform with the absence of p-hacking. This forms a testable null hypothesis and suggests statistical tests for p-hacking. We extend our results to phacking when there is also publication bias, and also consider what types of distributions arise under the alternative hypothesis that researchers engage in p-hacking. We show that the power of statistical tests for detecting p-hacking is low even if p-hacking is quite prevalent.

Detecting p‐Hacking

“…Therefore, we also show results based on de-rounded data. 18 In an earlier version of this paper (Elliott, Kudrin, and Wüthrich (2020)), we showed that de-rounding restores the non-increasingness but not the continuity of the p-curve. The right panel of Figure 4 shows the impact of derounding on the shape of the p-curve.…”