Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance

Pérignon, Christophe; Akmansoy, Olivier; Hurlin, Christophe; Dreber, Anna; Holzmeister, Felix; Huber, Juergen; Johannesson, Magnus; Kirchler, Michael; Menkveld, Albert J.; Razen, Michael; Weitzel, Utz

doi:10.2139/ssrn.4064172

Cited by 6 publications

(9 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Published results can be subject to bias towards positive outcomes and consequently researchers often seek to reproduce studies. Most of the time, perfect replicability is impossible because of data availability, interpretation leeway, and coding choices (see Pérignon et al (2023) for more on the matter). Nevertheless, even if coefficients or t-statistics cannot exactly be reproduced, it is useful to evaluate if their magnitudes are reasonable -and forking paths are the appropriate tool for this task.…”

Section: Corroborating Published Resultsmentioning

confidence: 99%

“…If it does, then the result can be generalized and sees its practical reach extended. Pérignon et al (2023) put forward two types of reanalyses: reproductions and replications. Reproduction occurs when the dataset is exactly the same as the original study, but the code can be different.…”

Section: Beyond Replication: Confirmationmentioning

confidence: 99%

“…1 See Huntington-Klein et al (2021), Breznau et al (2022), Gould et al (2023), Huber et al (2023) and the fincap study (exploited in Menkveld et al (2023) and Pérignon et al (2023)). Another enlightening exercise is provided by FivethirtyEight, in its Science is not broken article, in the political science discipline.…”

Section: Replication: a Necessity And A Challengementioning

confidence: 99%

See 2 more Smart Citations

Forking paths in empirical studies

Coqueret

2022

SSRN Journal

View full text Add to dashboard Cite

Section: Corroborating Published Resultsmentioning

confidence: 99%

Section: Beyond Replication: Confirmationmentioning

confidence: 99%

Section: Replication: a Necessity And A Challengementioning

confidence: 99%

See 1 more Smart Citation

Forking paths in empirical studies

Coqueret

2022

SSRN Journal

View full text Add to dashboard Cite

“…(2021), Chen (2021), Mitton (2022), Jensen, Kelly, and Pedersen (2023), and Pérignon et al. (2023). None of these replication studies focuses on explaining the dispersion of estimates in a cross‐section of researchers, or on the impact of peer feedback.…”

mentioning

confidence: 99%

“…The literature on replicability in finance is young but growing rapidly. Examples include McLean andPontiff (2016), Hou, Xue, andZhang (2018), Linnainmaa and Roberts (2018), Chordia, Goyal, and Saretto (2020), Harvey andLiu (2020), Ben-David, Franzoni, andMoussawi (2021), Black et al (2021), Chen (2021), Mitton (2022), Jensen, Kelly, andPedersen (2023), andPérignon et al (2023). 10 None of these replication studies focuses on explaining the dispersion of estimates in a cross-section of researchers, or on the impact of peer feedback.…”

mentioning

confidence: 99%

Nonstandard Errors

MENKVELD,

DREBER,

HOLZMEISTER

et al. 2024

The Journal of Finance

Self Cite

View full text Add to dashboard Cite

In statistics, samples are drawn from a population in a data‐generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence‐generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer‐review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.

show abstract

Reproduce to validate: A comprehensive study on the reproducibility of economics research

Herbert,

Kingi,

Stanchi

et al. 2024

Canadian J of Economics

View full text Add to dashboard Cite

Journals have pushed for transparency of research through data availability policies. Such data policies improve availability of data and code, but what is the impact on reproducibility? We present results from a large reproduction exercise for articles published in the American Economic Journal: Applied Economics, which has had a data availability policy since its inception in 2009. Out of 363 published articles, we assessed 274 articles. All articles provided some materials. We excluded 122 articles that required confidential or proprietary data or that required the replicator to otherwise obtain the data (44.5% of assessed articles). We attempted to reproduce 152 articles and were able to fully reproduce the results of 68 (44.7% of attempted reproductions). A further 66 (43.4% of attempted reproductions) were partially reproduced. Many articles required complex code changes even when at least partially reproduced. We collect bibliometric characteristics of authors, but find no evidence for author characteristics as determinants of reproducibility. There does not appear to be a citation bonus for reproducibility. The data availability policy of this journal was effective to ensure availability of materials, but is insufficient to ensure reproduction without additional work by replicators.

show abstract

Reproducibility of Empirical Results: Evidence from 1,000 Tests in Finance

Cited by 6 publications

References 51 publications

Forking paths in empirical studies

Forking paths in empirical studies

Nonstandard Errors

Reproduce to validate: A comprehensive study on the reproducibility of economics research

Contact Info

Product

Resources

About