Systematic review of the use of “magnitude-based inference” in sports science and medicine

Abstract

Magnitude-based inference (MBI) is a controversial statistical method that has been used in hundreds of papers in sports science despite criticism from statisticians. To better understand how this method has been applied in practice, we systematically reviewed 232 papers that used MBI. We extracted data on study design, sample size, and choice of MBI settings and parameters. Median sample size was 10 per group (interquartile range, IQR: 8-15) for multi-group studies and 14 (IQR: 10-24) for single-group studies; few studies reported a priori sample size calculations (15%). Authors predominantly applied MBI's default settings and chose "mechanistic/non-clinical" rather than "clinical" MBI even when testing clinical interventions (only 16 studies out of 232 used clinical MBI). Using these data, we can estimate the Type I error rates for the typical MBI study. Authors frequently made dichotomous claims about effects based on the MBI criterion of a "likely" effect and sometimes based on the MBI criterion of a "possible" effect. When the sample size is n = 8 to 15 per group, these inferences have Type I error rates of 12%-22% and 22%-45%, respectively. High Type I error rates were compounded by multiple testing: Authors reported results from a median of 30 tests related to outcomes; and few studies specified a primary outcome (14%). We conclude that MBI has promoted small studies, promulgated a "black box" approach to statistics, and led to numerous papers where the conclusions are not supported by the data. Amidst debates over the role of p-values and significance testing in science, MBI also provides an important natural experiment: we find no evidence that moving researchers away from p-values or null hypothesis significance testing makes them less prone to dichotomization or over-interpretation of findings.

Plyometric jump training effects on the physical fitness of individual-sport athletes: a systematic review with meta-analysis

“…Although two studies can be used in meta-analyses ( Valentine, Pigott & Rothstein, 2010 ), considering reduced sample sizes are common in the sport science literature ( Pigott, 2012 ), including in PJT studies ( Abt et al, 2020 ; Lohse et al, 2020 ; Ramirez-Campillo et al, 2018 , 2020c ), meta-analysis was only conducted when >3 studies were available ( Garcia-Hermoso, Ramirez-Campillo & Izquierdo, 2019 ; Moran, Ramirez-Campillo & Granacher, 2018 ; Skrede et al, 2019 ). Means and standard deviations (SD) for a measure of pre-post-intervention physical fitness from the PJT and control groups were converted to Hedges’ g effect size (ES).…”

The Effectiveness of Post-exercise Stretching in Short-Term and Delayed Recovery of Strength, Range of Motion and Delayed Onset Muscle Soreness: A Systematic Review and Meta-Analysis of Randomized Controlled Trials

“…It is possible to use two studies in a meta-analysis (Valentine et al, 2010), but we chose to establish a minimum of three studies (Moran et al, 2018;García-Hermoso et al, 2019;Skrede et al, 2019) to avoid small sample sizes (Abt et al, 2020;Lohse et al, 2020). Pre-and post-intervention means and standard deviations (SDs) were converted to Hedge's g effect size (ES) (García-Hermoso et al, 2019;Skrede et al, 2019).…”

Parastomal hernia after ileal conduit urinary diversion: re-visiting the predictors radiologically and according to patient-reported outcome measures

“…Where possible, this decision should be informed by existing literature and not arbitrarily chosen as representative of a 'small' effect. For example, evidence may or may not support a 'smallest effect size of interest' of +0.2, similar to that commonly used in MBI (Lohse et al, 2020). A 95% confidence interval that excludes a value (e.g., the smallest effect size of interest) implies that value can be rejected in a two-sided hypothesis test using an alpha of 0.05 (likewise a 90% confidence interval for a one-sided hypothesis test).…”