Dealing with the replication crisis in psychological science: The contribution of Type M and Type S errors

Abstract

The present work aims to analyze the replicability crisis in psychology with a focus on statistical inference. The main objective is to highlight the risks to beware when performing hypotheses tests in a Frequentist framework. In addition to the classic Type I and Type II errors, two other errors that are not commonly considered are Type M error (magnitude) and Type S error (sign), concerning the size and direction of the effects. The first chapter introduces Null Hypothesis Significance Testing (NHST), the prevalent approach to statistical inference in the social sciences, following a historical perspective and presenting also the approaches of the statisticians Ronald Fisher, Jerzy Neyman and Egon Pearson. The second chapter discusses the replicability crisis in psychology with an analysis of the origins, the factors that contributed to the crisis and solutions proposed for a change of direction. The third chapter analyzes the role of Type M and Type S errors in the replicability crisis. Studies with a high probability of committing these two types of errors could provide estimates of effects that are exaggerated and/or in the wrong direction. Two types of analysis are presented to examine these errors before conducting a study (prospective design analysis) or once the study has already been conducted (retrospective design analysis). The fourth chapter aims to link Type M and Type S errors with the decline effect, which is the observation that the magnitude of effects decreases with repeated replications. Although there may be multiple reasons behind the decline effect, a possible explanation is that the original study overestimated the effect. A case study illustrates how a retrospective design analysis of the original study can provide information on the probability of Type M and Type S errors and give support to the hypothesis of overestimation. The final chapter summarizes the contributions of Type M and Type S errors to the replication crisis and the role of a design analysis in planning studies and analyzing results.

A new method to explore inferential risks associated with each study in a meta-analysis: An approach based on Design Analysis

“…Given that the power of a study tells the likelihood of a statistical test to detect an effect if it truly exists, a consequence of underpower studies is that if the effect of interest actually exists, the study will have a low probability to find it (Altoè et al, 2020;Bertoldo, 2019). This leads to a high rates of false negative that reduces the reliability of psychological research and, combined with the pressure for authors to produce positive results, can ultimately cause damage to theory generation (Chambers, 2019).…”

“…In fact, the statistical significance could be the results of other factors rather than the presence of the true effect. For instance, researcher degrees of freedom, sampling and measurement errors and extraneous variables all contribute to inflate the probability of findings a p-value that is below the critical value (Altoè et al, 2020;Bertoldo, 2019).…”

“…Moreover, it requires authors to adhere to the hypotetico-deductive model by think carefully about all phases of the research process and about some aspects that sometimes are ignored or forgotten, such as whether or not issue could arise and how to solve them and if the claims they want to make have the capability of being falsified (Chambers, 2019;Lakens, 2019). In fact, planning the research project carefully is crucial to be able to make accurate decisions regarding, for example, the sample size necessary to detect an effect size of interest (Bertoldo, 2019). The key rule of planning a research project will be examined further in the next chapter, when will be described the Design Analysis.…”

“…It is important to note that this inferential risk is not a probability but rather a ratio between the absolute value of the mean of all the effect sizes that are statistically significant and the plausible true effect size. And it indicates the average percentage of inflation (Bertoldo, 2019). Consequently, if there is no exaggeration of the effect that emerges as statistically significant, the value that is obtained is one (i.e., minimum obtainable value).…”

“…It is calculated by making the ratio between the probability to find a negative statistically effect size and the power of a test, when the plausible effect size is positive. Otherwise, the ratio is between the power of a test and the probability to find a positive statistically significant effect (Bertoldo, 2019). Furthermore, both this errors are strictly related to power.…”