Meta-Analytic Findings in the Self-Controlled Motor Learning Literature: Underpowered, Biased, and Lacking Evidential Value

Abstract

The self-controlled motor learning literature consists of experiments that compare a group of learners who are provided with a choice over an aspect of their practice environment to a group who are yoked to those choices. A qualitative review of the literature suggests an unambiguous benefit from self-controlled practice. A meta-analysis was conducted on the effects of self-controlled practice on retention test performance measures with a focus on assessing and potentially correcting for selection bias in the literature, such as publication bias and p-hacking. First, a naïve random effects model was fit to the data and a moderate benefit of self-controlled practice, g=.44 (k= 52,N= 3134,95%CI[.31, .56]), was found. Second, publication status was added to the model as a potential moderator, revealing a significant difference between published and unpublished findings, with only the former reporting a benefit of self-controlled practice. Third, to investigate and adjust for the impact of selectively reporting statistically significant results, a weight-function model was fit to the data with a one-tailed p-value cutpoint of .025. The weight-function model revealed substantial selection bias and estimated the true average effect of self-controlled practice as g=.107 (95%CI[.047, .18]). P-curve analyses were conducted on the statistically significant results published in the literature and the outcome suggested a lack of evidential value. Fourth, a suite of sensitivity analyses were conducted to evaluate the robustness of these results, all of which converged on trivially small effect estimates. Overall, our results suggest the benefit of self-controlled practice on motor learning is small and not currently distinguishable from zero.

Increased perceptions of autonomy through choice over one's observation schedule fail to enhance motor skill retention

“…In our experiment, the estimated effect size of self-controlled versus yoked (collapsed across our Traditional and Explicit Yoked groups) was g = 0.22. While this estimate is larger than those from the weight-function model and PEESE method, it does fall within the range of plausible effect sizes (McKay et al, 2021). When the lack of a self-controlled advantage in previous work and our current experiment are contextualized within the findings of this recent meta-analysis, it is not as surprising that the so-called self-controlled learning benefit was not replicated.…”

“…Ultimately, our results add to a growing body of evidence that questions whether autonomy-supportive manipulations directly affect learning. One explanation for this lack of replicability is that the "true" effect of such manipulations are much smaller than previously estimated (McKay et al, 2021). Given the resources required (in terms of sample size) to reliably detect the tiny effects, we encourage motor learning scientists to invest their limited resources carefully-either studying practice conditions that have much bigger effects or devising ways to reduce variability and increase power through clever experimental design (McClelland, 2000).…”

“…This recent lack of support may arise from the selfcontrolled learning advantage being a much smaller effect than originally estimated (g = 0.63 by McKay et al 2014). A more recent estimate of this effect from a meta-analysis using a weight-function model was g = .11, 95%CI = [.047, .18] McKay et al, 2021). This estimate was further reduced to g = .054 after controlling for publication bias using the precision-effect estimate with standard error (PEESE) method.…”

“…Using the upper bound, g = .26, of the range of plausible effects, 253 participants are required to have 80% power to detect this effect in retention using an independent-samples t-test. The number of participants jumps to 1300 per group, if for instance the estimate from the weight-function model, g = .11, is accurate (McKay et al, 2021). Given the field of motor learning suffers from a lack of adequately powered designs (Lohse et al, 2016) and that underpowered designs are more likely to produce false positives and overestimated effect sizes (Button et al, 2013), we are skeptical of the replicability of the previous overwhelming support for the self-controlled learning advantage.…”

Meta-analysis of the reduced relative feedback frequency effect on motor learning and performance

“…the probability that any given study that has been conducted (published or not) would produce a significant result, termed the expected discovery rate (EDR). The delayed retention time point was considered the primary target for a significant finding in motor learning research and previous research has found substantial selection at that time point (McKay, Yantha, Hussien, Carter, & Ste-Marie, 2021). As such, we focused both the RoBMA and z-curve analyses on delayed retention results.…”