Designing Studies and Evaluating Research Results: Type M and Type S Errors for Pearson Correlation Coefficient

Abstract

It is widely appreciated that many studies in psychological science suffer from low statistical power. One of the consequences of analyzing underpowered studies with thresholds of statistical significance, is a high risk of finding exaggerated effect size estimates, in the right or in the wrong direction. These inferential risks can be directly quantified in terms of Type M (magnitude) error and Type S (sign) error, which directly communicate the consequences of design choices on effect size estimation. Given a study design, Type M error is the factor by which a statistically significant effect is on average exaggerated. Type S error is the probability to find a statistically significant result in the opposite direction to the plausible one. Ideally, these errors should be considered during a prospective design analysis in the design phase of a study to determine the appropriate sample size. However, they can also be considered when evaluating studies’ results in a retrospective design analysis. In the present contribution we aim to facilitate the considerations of these errors in the research practice in psychology. For this reason we illustrate how to consider Type M and Type S errors in a design analysis using one of the most common effect size measures in psychology: Pearson correlation coefficient. We provide various examples and make the R functions freely available to enable researchers to perform design analysis for their research projects.

The comparison of two independent proportions via Design Analysis: Theoretical background and an R implementation

“…Type S error is the probability of finding a significant effect in the wrong direction, in other word, an effect with the opposite sign to the plausible effect size (Gelman and Carlin, 2014;Altoè et al, 2020;Bertoldo, Callagher and Altoè, 2021). To estimate these parameters, scientists need to use a plausible effect size, but let's see first how they are computed (Bertoldo et al, 2021). Reprinted from Gelman and Carlin, 2004).…”

“…probability to find a result in the wrong direction (Gelman and Carlin, 2014). These errors are additional information to statistical power, and both increase when sample size and/or effect size decrease (in are making it more difficult to find a result in the wrong direction, type M error increases because to find a significant effect with a lower p-value (maintaining sample size) the effect size has to be exaggerated even more (Bertoldo et al, 2021).…”

“…The study on which we will do a design analysis is a study that has already been cited in the First Chapter. Rather than taking a research without power analysis, as it has already been done several times (Gelman and Carlin, 2014;Bertoldo, 2019;Altoè et al, 2020;Bertoldo et al, 2021) we will apply design analysis to see if the results are comparable to power calculations conducted in the original study, to add information about type M and type S errors and to evaluate if retrospective design analysis on results is coherent with prospective design analysis. The research we will look at is "No Replication, No Trust?…”

A new method to explore inferential risks associated with each study in a meta-analysis: An approach based on Design Analysis

“…On the other hand, Type S error represents the probability that a statistically significant result has the opposite sign of the plausible true effect size. This analysis is called Design Analysis, precisely to emphasize the importance of both the study design Bertoldo et al, 2020;Gelman & Carlin, 2014) and the role of estimates and uncertainties in making more reasonable statistical claims (Altoè et al, 2020;Gelman & Carlin, 2014).…”

“…In fact, even though design analysis should be ideally performed when designing a study, as a simple size planning strategy (i.e., prospective design analysis), it can also be efficiently used to evaluate the already obtained study results and the associated inferential risk (i.e., retrospective design analysis) (Altoè et al, 2020;Bertoldo et al, 2020). Furthermore, design analysis reserves particular attention to the identification of the plausible magnitude and direction of the effect under study, refers to as plausible effect size.…”