Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics

Brodeur, Abel; Cook, Nikolai; Heyes, Anthony

doi:10.1257/aer.20190687

Cited by 233 publications

(204 citation statements)

References 40 publications

Supporting

Mentioning

192

Contrasting

Unclassified

Order By: Relevance

“…The insight concerns publication bias, the potential tendency of authors, editors, and referees to prefer results that are statistically significant and consistent with previous findings or underlying theory. The bias has been discussed, among others, by Havranek (2015), Brodeur et al (2016), Bruns & Ioannidis (2016), Christensen & Miguel (2018), Brodeur et al (2020), and Blanco-Perez & . Ioannidis et al (2017) show that publication bias looms large in economics and finance, exaggerating the mean reported coefficient twofold.…”

Section: Differences Estimates Levels Estimatesmentioning

confidence: 97%

How puzzling is the forward premium puzzle? A meta-analysis

Žigraiová

Havránek

Iršová

et al. 2021

European Economic Review

View full text Add to dashboard Cite

A key theoretical prediction in financial economics is that under risk neutrality and rational expectations a currency's forward rates should form unbiased predictors of future spot rates. Yet scores of empirical studies report negative slope coefficients from regressions of spot rates on forward rates, which is inconsistent with the forward rate unbiasedness hypothesis. We collect 3,643 estimates from 91 research articles and using recently developed techniques investigate the effect of publication and misspecification biases on the reported results. Correcting for these biases we estimate the slope coefficients of 0.31 and 0.98 for developed and emerging currencies respectively, which implies that empirical evidence is in line with the theoretical prediction for emerging economies and less puzzling than commonly thought for developed economies. Our results also suggest that the coefficients are systematically influenced by the choice of data, numeraire currencies, and estimation methods.

show abstract

Section: Differences Estimates Levels Estimatesmentioning

confidence: 97%

How puzzling is the forward premium puzzle? A meta-analysis

Žigraiová

Havránek

Iršová

et al. 2021

European Economic Review

View full text Add to dashboard Cite

show abstract

“…Using data from Brodeur et al (2016), Coffman and Niederle (2015) argue that there is little evidence of p-hacking in experimental work. In fact, the type of studies where Brodeur et al (2016) and Brodeur et al (2020) find evidence of p-hacking is in non-experimental work that relies on instrumental variables and other methods of causal identification. Coffman and Niederle (2015) point out that those championing the use of pre-analysis plans focus almost exclusively on their use in RCTs, the place where, given available data, they are needed the least.…”

Section: Reconciling Costs and Benefits Of Pre-analysis Plans In Economicsmentioning

confidence: 99%

Ulysses' pact or Ulysses' raft: Using pre‐analysis plans in experimental and nonexperimental research

Janzen

Michler

2021

Applied Eco Perspectives Pol

View full text Add to dashboard Cite

Economists have recently adopted pre-analysis plans in response to concerns about robustness and transparency in research. The increased use of registered pre-analysis plans has raised competing concerns that detailed plans are costly to create, overly restrictive, and limit the type of inspiration that stems from exploratory analysis. We consider these competing views of pre-analysis plans, and make a careful distinction between the roles of pre-analysis plans and registries, which provide a record of all planned research. We propose a flexible "packraft" pre-analysis plan approach that offers benefits for a wide variety of experimental and nonexperimental applications in applied economics.

show abstract

“…For example, Brodeur et al (2016) use the term inflation and define it as a residual from a technical decomposition (the z-curve method). Brodeur et al (2020) use the more loaded term p-hacking. The intentions behind these other terms and definitions, as we understand them, are the same as ours.…”

Section: Researcher Bias Is About Undisclosed Exaggerationmentioning

confidence: 99%

Star Wars at Central Banks

Gorajek¹,

Bank²,

Staib³

et al. 2021

Preprint

View full text Add to dashboard Cite

We investigate the credibility of central bank research by searching for traces of researcher bias, which is a tendency to use undisclosed analytical procedures that raise measured levels of statistical significance (stars) in artificial ways. To conduct our search, we compile a new dataset and borrow 2 bias-detection methods from the literature: the p-curve and z-curve. The results are mixed. The p-curve shows no traces of researcher bias but has a high propensity to produce false negatives. The z-curve shows some traces of researcher bias but requires strong assumptions. We examine those assumptions and challenge their merit. At this point, all that is clear is that central banks produce results with patterns different from those in top economic journals, there being less bunching around the 5 per cent threshold of statistical significance.

show abstract

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics

Cited by 233 publications

References 40 publications

How puzzling is the forward premium puzzle? A meta-analysis

How puzzling is the forward premium puzzle? A meta-analysis

Ulysses' pact or Ulysses' raft: Using pre‐analysis plans in experimental and nonexperimental research

Star Wars at Central Banks

Contact Info

Product

Resources

About