Methods Matter: P-Hacking and Causal Inference in Economics

Abstract

The economics 'credibility revolution' has promoted the identification of causal relationships using difference-in-differences (DID), instrumental variables (IV), randomized control trials (RCT) and regression discontinuity design (RDD) methods. The extent to which a reader should trust claims about the statistical significance of results proves very sensitive to method. Applying multiple methods to 13,440 hypothesis tests reported in 25 top economics journals in 2015, we show that selective publication and phacking is a substantial problem in research employing DID and (in particular) IV. RCT and RDD are much less problematic. Almost 25% of claims of marginally significant results in IV papers are misleading.

Detecting p-hacking

“…We can see that, depending on π, the t-curve takes many different forms. This simple numerical example further demonstrates that the distribution of alternatives can induce humps around 1.96, as documented empirically for instance by Gerber and Malhotra (2008), Brodeur et al (2016Brodeur et al ( , 2018 and Vivalt (2019), even if there is no phacking. Thus, humps generated by p-hacking cannot be distinguished from humps generated by the distribution of alternatives, which suggests that testing for p-hacking based on the shape of the t-curve around 1.96 (or any other significance threshold) can be problematic.…”

“…A researchers' ability to explore various ways of analyzing and manipulating data and then selectively report the ones that yield statistically significant results, commonly referred to as p-hacking, undermines the scientific credibility of reported results. To assess the extent of this problem researchers have begun to examine distributions of t-statistics (t-curves) and p-values (p-curves) across studies in various fields, with mixed results (e.g., Bishop and Thompson, 2016;Brodeur et al, 2016Brodeur et al, , 2018Gerber and Malhotra, 2008;Head et al, 2015;Jager and Leek, 2013;Simonsohn et al, 2014;Vivalt, 2019). This paper examines more closely what can be learned from these distributions, and whether or not these tests are likely to be informative about the extent to which p-hacking occurs.…”

Relative Risk Aversion: A Meta-Analysis

“…Note that publication bias and p-hacking are observationally equivalent, so for parsimony we will use the term publication bias to describe both, as is common in the meta-analysis literature. Many studies have recently discussed how publication bias can exaggerate empirical estimates in economics (Brodeur et al, 2016;Bruns & Ioannidis, 2016;Card et al, 2018;Christensen & Miguel, 2018;DellaVigna et al, 2019;Blanco-Perez & Brodeur, 2020;Brodeur et al, 2020;Ugur et al, 2020;Xue et al, 2020;Neisser, 2021;Stanley et al, 2021;DellaVigna & Linos, 2022;Stanley et al, 2022), and the exaggeration can be twofold or more (Ioannidis et al, 2017). Publication bias is natural, common in economics, and does not imply cheating or any ulterior motives on the part of the researchers.…”

Hedge Fund Performance: A Quantitative Survey

“…Selective publication may result from both conscious and subconscious decisions made by authors, editors, and referees who discard results that look implausible in the light of their a priori expectations (Ioannidis et al, 2017). Publication selection bias and its implications are extensively discussed in prior literature including Stanley (2001Stanley ( , 2005; ; Havranek (2015); Brodeur et al (2016); Bruns & Ioannidis (2016); ; Christensen & Miguel (2018); Brodeur et al (2020a); Blanco-Perez & Brodeur (2020); Zigraiova et al (2021). These studies document that publication selection bias is indeed widespread in a wide range of economic settings and it has a substantial impact on the mean value of reported estimates.…”