Multiple Perspectives on Inference for Two Simple Statistical Scenarios

Abstract

When data analysts operate within different statistical frameworks (e.g., frequentist versus Bayesian, emphasis on estimation versus emphasis on testing), how does this impact the qualitative conclusions that are drawn for real data? To study this question empirically we selected from the literature two simple scenarios --involving a comparison of two proportions and a Pearson correlation-- and asked four teams of statisticians to provide a concise analysis and a qualitative interpretation of the outcome. The results showed considerable overall agreement; nevertheless, this agreement did not appear to diminish the intensity of the subsequent debate over which statistical framework is more appropriate to address the questions at hand.

Seven steps toward more transparency in statistical practice

“…The disagreement also manifests itself in practical application whenever multiple statisticians and practitioners of statistics find themselves independently analysing the same dataset. Specifically, recent 'multiple-analyst' articles show that statisticians rarely use the same analysis, and they often draw different conclusions, even for the exact same dataset and research question [3][4][5][6][7] . Deep disagreement is also exhibited by contradictory guidelines on P values [8][9][10][11][12][13] .…”

Scientific self-correction: the Bayesian way

“…In this paper, we shall not enter into the foundational debate between Bayesians and frequentists (for surveys of arguments, see, e.g., Romeijn 2014;Sprenger 2006;Mayo 2018;van Dongen et al 2019). We just note that while Bayesian inference avoids the typical problems of frequentist inference with NHST, it is not exempt from limitations.…”

Calibrating the Scientific Ecosystem Through Meta-Research

“…Others suggest a switch to alternative paradigms, such as Bayesian statistics, which has several advantages (Wagenmakers, 2007). However, it is noteworthy that different inferential approaches often arrive at similar conclusions, at least in simple scenarios ( van Dongen et al, 2019). Furthermore, the success of any given approach is largely dependent on the capabilities of the researchers conducting the statistical analyses.…”