Detecting p‐Hacking

Abstract

We theoretically analyze the problem of testing for p‐hacking based on distributions of p‐values across multiple studies. We provide general results for when such distributions have testable restrictions (are non‐increasing) under the null of no p‐hacking. We find novel additional testable restrictions for p‐values based on t‐tests. Specifically, the shape of the power functions results in both complete monotonicity as well as bounds on the distribution of p‐values. These testable restrictions result in more powerful tests for the null hypothesis of no p‐hacking. When there is also publication bias, our tests are joint tests for p‐hacking and publication bias. A reanalysis of two prominent data sets shows the usefulness of our new tests.

Forking paths in empirical studies

“…With forking paths, we should not be interested in p-hacking tests such as the ones in Elliott et al (2022) because the purpose of exhaustive paths is precisely to avoid p-hacking in the first place. However, out of curiosity and in full disclosure, we have used these tests on the series of paths obtained from our two empirical studies and got disappointing results: while we did not hack, the tests in some cases concluded that we did.…”

“…In Elliott et al (2022), there are 3 important distributions: F h , the one of the test statistic, F the distribution that determines the critical values under the null, and Π, the true distribution of the effect under scrutiny. The latter does not matter, as all tests in Elliott et al (2022) hold irrespective of Π. Critical values are also rarely an issue, as they are often determined based on standard asymptotic results.…”

“…The research community has also proposed other avenues to tackle these issues, e.g., by identification and detection methods for false discoveries and p-hacking (Simonsohn et al (2014a,b), Elliott et al (2022)), and even solutions via model averaging (Moral-Benito (2015), Steel (2020)), extreme bound analysis (Leamer and Leonard (1983), Granger and Uhlig (1990)), correction measures (Andrews and Kasy (2019)), shrinkage (van Zwet and Cator (2021)), robust aggregation (Rytchkov and Zhong (2020)), noise dissemination (Echenique and He (2023)), new critical values (McCloskey and Michaillat (2023)), specification curve analysis (Simonsohn et al (2020)), or Bayesian publication decisions (Frankel and Kasy (2022)).…”

“…Finally, we mention the ultimate level of snooping, which we call vicious p-hacking. This refers to the case when paths are explored until a sufficient amount of p-values can be extracted in such a way that they successfully pass a test that detects p-hacking, for instance those mentioned in Elliott et al (2022). Because such tests are very recent, it is highly unlikely that vicious p-hacking has already occurred.…”

“…Suppose we focus on the common case of one-sided t-tests, which we will cover in Section 4 for anomaly detection. In Elliott et al (2022), it is shown in Theorem 2 that, under mild technical assumptions and in the absence of p-hacking, the density of the pvalues should be decreasing and convex. Because of their importance, discuss the relevance and applicability of p-hacking tests for forking paths in Appendix C.…”