Small sample mediation testing: Misplaced confidence in bootstrapped confidence intervals.

Abstract

Bootstrapping is an analytical tool commonly used in psychology to test the statistical significance of the indirect effect in mediation models. Bootstrapping proponents have particularly advocated for its use for samples of 20-80 cases. This advocacy has been heeded, especially in the Journal of Applied Psychology, as researchers are increasingly utilizing bootstrapping to test mediation with samples in this range. We discuss reasons to be concerned with this escalation, and in a simulation study focused specifically on this range of sample sizes, we demonstrate not only that bootstrapping has insufficient statistical power to provide a rigorous hypothesis test in most conditions but also that bootstrapping has a tendency to exhibit an inflated Type I error rate. We then extend our simulations to investigate an alternative empirical resampling method as well as a Bayesian approach and demonstrate that they exhibit comparable statistical power to bootstrapping in small samples without the associated inflated Type I error. Implications for researchers testing mediation hypotheses in small samples are presented. For researchers wishing to use these methods in their own research, we have provided R syntax in the online supplemental materials.

Why might poor sleep quality lead to depression? A role for emotion regulation

“…Specifically, the sample was collected originally for a study of autonomic nervous system functioning in depression and sample sizes reflected the requirements of the parent study (see Salomon et al, 2013 for details of recruitment and screening procedures); the current sample size, however, was also adequate to assess mediation effects using bootstrapping (Koopman et al, 2014). Community participants were recruited and visited the laboratory for clinical and psychophysiological assessments, as well as completing self-report scales.…”

Power in Bayesian Mediation Analysis for Small Sample Research

“…Several frequentist methods for mediation analysis have been designed to deal with issues that arise in small samples (Hoyle & Kenny, 1999), and Bayesian methods have been proposed as an alternative to frequentist methods with small samples for SEM (Hox, van de Schoot, & Matthijsse, 2012; Lee & Song, 2004), growth mixture models (Depaoli, 2013), and multilevel models (McNeish, 2016a). It was suggested that Bayesian methods have potential for increasing power in mediation analysis (Koopman, Howe, Hollenbeck, & Sin, 2015; Yuan & MacKinnon, 2009), however, this is the first paper to focus on the potential of Bayesian methods with diffuse and informative conjugate priors for mediation analysis in small samples. The first goal of this study is to examine the extent to which using prior information in a Bayesian analysis with small samples increases the power to detect the mediated effect compared to commonly used frequentist methods and Bayesian methods with diffuse priors.…”

“… It was suggested that Bayesian methods have potential for increasing power in mediation analysis (Koopman, Howe, Hollenbeck, & Sin, 2015; Yuan & MacKinnon, 2009). This paper compares the power of Bayesian credibility intervals for the mediated effect to the power of normal theory, distribution of the product, percentile, and bias-corrected bootstrap confidence intervals at N≤ 200.…”

The Role of Psychosocial Processes in the Development and Maintenance of Chronic Pain

“…Increasingly, mediation studies in the field have utilized bootstrapping, which is a general approach to statistical inference based on random re-sampling from the observed data. In general, the traditional Baron and Kenny 14 regression-based mediation analyses has fairly limited power; bootstrapping has been widely recommended as it improves power, even in relatively small samples, and obviates concerns over violating normality assumptions 17;124 .…”