Enhancing Statistical Inference in Psychological Research via Prospective and Retrospective Design Analysis

Abstract

In the past two decades, psychological science has experienced an unprecedented replicability crisis which uncovered several problematic issues. Among others, the use and misuse of statistical inference plays a key role in this crisis. Indeed, statistical inference is too often viewed as an isolated procedure limited to the analysis of data that have already been collected. Instead, statistical reasoning is necessary both at the planning stage and when interpreting the results of a research project. Based on these considerations, we build on and further develop an idea proposed by Gelman and Carlin (2014) termed "prospective and retrospective design analysis".Rather than focusing only on the statistical significance of a result and on the classical control of type I and type II errors, a comprehensive design analysis involves reasoning about what can be considered a plausible effect size. Furthermore, it introduces two relevant inferential risks: the exaggeration ratio or Type M error (i.e., the predictable average overestimation of an effect that emerges as statistically significant), and the sign error or Type S error (i.e., the risk that a statistically significant effect is estimated in the wrong direction). design analysis is that it can be usefully carried out both in the planning phase of a study and for the evaluation of studies that have already been conducted, thus increasing researchers awareness during all phases of a research project. To illustrate the benefits of design analysis to the widest possible audience, we use a familiar example in psychology where the researcher is interested in analyzing the differences between two independent groups considering Cohens d as an effect size measure. We examine the case in which the plausible effect size is formalized as a single value, and propose a method in which uncertainty concerning the magnitude of the effect is formalized via probability distributions. Through several examples and an application to a real case study, we show that even though a design analysis requires big effort, it has the potential to contribute to planning more robust and replicable studies. Finally, future developments in the Bayesian framework are discussed. Keywordsprospective and retrospective design analysis, Type M and Type S errors, effect size, power, psychological research, statistical inference, statistical reasoning, R functions "If statisticians agree on one thing, it is that scientific inference should not be made mechanically." Gigerenzer and Marewski (2015, p. 422) "Accept uncertainty. Be thoughtful, open, and modest.

Dyslexia treatment studies: A systematic review and suggestions on testing treatment efficacy with small effects and small samples

“…The latter not only makes it difficult to distinguish true results from false positive results, but also inflates the risk of overestimating effect sizes. This risk can be defined a priori as the "exaggeration ratio" that indicates how much an effect size will be overestimated on the average in comparison with a plausible true effect size given that statistical significance is reached (e.g., Gelman & Carlin, 2014; see also Altoè et al, 2020). Unfortunately, researchers who test treatment efficacy in learning disorders frequently encounter this problem.…”

Dealing with the replication crisis in psychological science: The contribution of Type M and Type S errors

“…This might not accurately reflect researchers' expectations which can instead be better captured by an interval of values with an associated probability distribution. This suggestion is currently under development in the work of Altoè et al (2019). Finally, in the current work, we chose Cohen's d as an effect size measure to illustrate a design analysis because it is an effect size measure widely used in psychology.…”

“…Power, Type M error and Type S error as a function of sample size in an independent samples t-test, assuming a plausible effect size equal to a Cohen's d of 0.35 and  of 0.05.Reprinted from 'Enhancing statistical inference in psychological research via prospective and retrospective design analysis.' by G Altoè et. al, 2019..…”

Using Mobile Devices in Teaching Large University Classes: How Does It Affect Exam Success?

“…The course lasts 42 h (11 weeks, 21 lessons, with two weekly sessions lasting 90 min each). Since we could not plan the number of participants in advance, we ran a retrospective power analysis via simulation on the number of participants collected to test the reliability of our results ( Altoè et al, 2020 ). Hypothesizing a plausible correlation of 0.30 between exam results and the use of quizzes and out-of-class activities ( Castillo-Manzano et al, 2016 ; Hunsu et al, 2016 ; Sung et al, 2016 ; Lei et al, 2018 ), 294 participants yielded a power of 0.84, with a plausible magnitude error of 0.72 and no sign error.…”