Reporting errors and biases in published empirical findings: Evidence from innovation research

Bruns, Stephan B.; Asanov, Igor; Bode, Rasmus; Dunger, Melanie; Funk, Christoph; Hassan, Sherif; Hauschildt, Julia; Heinisch, Dominik P.; Kempa, Karol; König, Johannes; Lips, J.; Verbeck, Matthias; Wolfschütz, Eva; Buenstorf, Guido

doi:10.1016/j.respol.2019.05.005

“…This dichotomous picture (which we relax when we extend the model) is in line with scientometric evidence for the increasing prevalence of statistically significant over non-significant findings (Fanelli 2012). The choice of 5% as a cutoff level is a well-entrenched convention in the behavioral sciences; that said, also "marginally significant" p-values (i.e., .05 ≤ p < .10 ) are often reported in economics and the biomedical sciences (De Winter and Dodou 2015;Lakens 2015;Bruns et al 2019).…”

Section: Variable 3: Suppressing Inconclusive Evidencesupporting

confidence: 69%

Scientific self-correction: the Bayesian way

Romero

¹

,

Sprenger

²

2020

View full text Add to dashboard Cite

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure-replacing null hypothesis significance testing (NHST) with Bayesian inference-in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study.

show abstract

“…This dichotomous picture (which we relax when we extend the model) is in line with scientometric evidence for the increasing prevalence of statistically significant over non-significant findings (Fanelli, 2012). The choice of 5% as a cutoff level is a well-entrenched convention in the behavioral sciences; that said, also "marginally significant" p-values (that is, .05 ≤ p < .10) are often reported in economics and the biomedical sciences (De Winter and Dodou, 2015;Lakens, 2015;Bruns et al, 2019).…”

Section: Variable 3: Suppressing Inconclusive Evidencesupporting

confidence: 68%

Scientific Self-Correction: The Bayesian Way

Romero¹,

Sprenger²

2019

Preprint

View full text Add to dashboard Cite

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure-replacing null hypothesis significance testing (NHST) with Bayesian inference-in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study.

show abstract

“…The reliability of empirical research is subject to intensive debate (e.g., Wasserstein and Lazar, 2016; Munafò et al ., 2017), with economics being no exception (e.g., Brodeur et al ., 2016; Chang and Li, 2017; Doucouliagos et al ., 2018; Bruns et al ., 2019; Vivalt, 2019). Most prominently, Ioannidis et al .…”

Section: Introductionmentioning

confidence: 99%

“…Our sample mainly comprises hypothesis‐testing regression coefficients that address the research question(s) of the respective article and they are typically reported with their respective standard errors and eye‐catchers to denote the respective levels of statistical significance. We use an algorithm that flags tests as potential reporting errors and gives authors the benefit of the doubt if rounding may be responsible for apparent reporting errors (e.g., Nuijten et al ., 2016; Bruns et al ., 2019). We verify the flagged tests by contacting all authors of the afflicted studies.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The (Non‐)significance of Reporting Errors in Economics: Evidence From Three Top Journals

Pütz

¹

,

Bruns

²

2020

Journal of Economic Surveys

Self Cite

View full text Add to dashboard Cite

We investigate the prevalence and sources of reporting errors in 30,993 hypothesis tests from 370 articles in three top economics journals. We define reporting errors as inconsistencies between reported significance levels by means of eye‐catchers and calculated p‐values based on reported statistical values, such as coefficients and standard errors. While 35.8% of the articles contain at least one reporting error, only 1.3% of the investigated hypothesis tests are afflicted by reporting errors. For strong reporting errors for which either the eye‐catcher or the calculated p‐value signals statistical significance but the respective other one does not, the error rate is 0.5% for the investigated hypothesis tests corresponding to 21.6% of the articles having at least one strong reporting error. Our analysis suggests a bias in favor of errors for which eye‐catchers signal statistical significance but calculated p‐values do not. Survey responses from the respective authors, replications, and exploratory regression analyses indicate some solutions to mitigate the prevalence of reporting errors in future research.

show abstract

Reporting errors and biases in published empirical findings: Evidence from innovation research

Cited by 35 publications

References 42 publications

Scientific self-correction: the Bayesian way

Scientific self-correction: the Bayesian way

Scientific Self-Correction: The Bayesian Way

The (Non‐)significance of Reporting Errors in Economics: Evidence From Three Top Journals

Contact Info

Product

Resources

About