Scientific self-correction: the Bayesian way

Abstract

The enduring replication crisis in many scientific disciplines casts doubt on the ability of science to estimate effect sizes accurately, and in a wider sense, to self-correct its findings and to produce reliable knowledge. We investigate the merits of a particular countermeasure-replacing null hypothesis significance testing (NHST) with Bayesian inference-in the context of the meta-analytic aggregation of effect sizes. In particular, we elaborate on the advantages of this Bayesian reform proposal under conditions of publication bias and other methodological imperfections that are typical of experimental research in the behavioral sciences. Moving to Bayesian statistics would not solve the replication crisis single-handedly. However, the move would eliminate important sources of effect size overestimation for the conditions we study.

The current phase of social sciences research: A thematic overview of the literature

“…An essential feature of science is self-correction. The long-run scientific method guarantees convergence to true theories as it enables the identification and correction of errors in published research (Romero & Sprenger, 2020). Publication bias has the potential to hinder science from achieving several elements of the process, such as replication of studies (Andrews & Kasy, 2019;Editorial (2019Editorial ( , 2019.…”

A Battle in the Statistics Wars: a simulation-based comparison of Bayesian, Frequentist and Williamsonian methodologies

“…Within philosophy, we found only two earlier studies comparing Bayesian and frequentist methodologies using simulations. Felipe Romero and Jan Sprenger employ simulations to argue in favour of a Bayesian approach to the replication crisis (Romero and Sprenger, 2020). Their study is interesting, but its topic is inferential success under different sorts of scientific institutions, and therefore very different from our own.…”

“…If there is any suspicion that philosophy of science is an ivory tower subject, it should be extinguished by what Deborah Mayo has called the "Statistics Wars" between classical statisticians, Bayesians, and a prismatic assortment of variations of these views (Ioannidis, 2005;Howson and Urbach, 2006;Wasserstein and Lazar, 2016;Mayo, 2018;van Dongen et al, 2019;Sprenger and Hartmann, 2019;Romero and Sprenger, 2020). Even apparently recondite questions about concepts like evidence, probability, and rational belief are connected with questions of statistical practice.…”