Low prevalence of a priori power analyses in motor behavior research

McKay, Brad; Corson, Abbey; Vinh, Mary-Anne; Jeyarajan, Gianna; Tandon, Chitrini; Brooks, Hugh; Hubley, Julie; Mj, Carter

doi:10.51224/srxiv.175

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lakens (2022) describes several approaches for determining sample size, including conducting an a priori power analysis. A recent survey of three popular motor learning journals revealed a low prevalence (84/635 or 13% in McKay, Corson, et al, 2022) and low reproducibility (7/84 or 8% in McKay, Bacelar, et al, 2022) of reported a priori power analyses. The usefulness of an a priori power analysis depends on reasonable assumptions about effect sizes.…”

Section: Discussionmentioning

confidence: 99%

The combination of reporting bias and underpowered study designs has substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: a meta-analysis

McKay

Bacelar

Parma

et al. 2023

International Review of Sport and Exercise Psychology

View full text Add to dashboard Cite

Enhanced expectancies and autonomy-support through self-controlled practice conditions form the motivation pillar of OPTIMAL theory (Wulf & Lewthwaite, 2016). The influence of these practice variables on motor learning was recently evaluated in two separate meta-analyses. Both meta-analyses found that the published literature suggested a moderate and significant benefit on motor learning; however, evidence for reporting bias was found in both literatures. Although multiple bias-corrected estimates were reported in the self-controlled meta-analysis, there was no principled way to prefer one over the other. In the enhanced expectancies meta-analysis, the trim-and-filltechnique failed to correct the estimated effects. Here, we addressed these limitations by reanalyzing the data from both meta-analyses using robust Bayesian meta-analysis methods. Our reanalysis revealed that reporting bias substantially exaggerated the benefits of these practice variables in the original meta-analyses. The true effects are instead small, uncertain, and potentially null. We also found the estimated average statistical power among all studies from the original meta-analyses was 6% (95% confidence interval [5%, 13%]). These results provide compelling and converging evidence that strongly suggests the available literature is insufficient to support the motivation pillar of OPTIMAL theory. Our results highlight the need for adequately powered experimental designs if motor learning scientists want to make evidence-based recommendations.

show abstract

Section: Discussionmentioning

confidence: 99%

The combination of reporting bias and underpowered study designs has substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: a meta-analysis

McKay

Bacelar

Parma

et al. 2023

International Review of Sport and Exercise Psychology

View full text Add to dashboard Cite

show abstract

“…In practice, NHST is typically executed by comparing the observed p-value (which is always provided using standard statistical software) to a pre-specified alpha level: the null hypothesis is rejected if the p-value is smaller than alpha and the null hypothesis is accepted if the pvalue is bigger than alpha 1 . Thus, in practice the beta-level is often ignored (McKay et al, 2022). The current paper focusses on the question how to specify the alpha and beta levels in NHST.…”

Section: On a Conditional Justification Of The Error Rates In Classic...mentioning

confidence: 99%

“…In current statistical practice, however, researchers often rely on the default (but arbitrary) choice of an alpha level of 0.05 and omit a power analysis (McKay et al, 2022), and thus the actual type II error rate is generally unknown. The default choice of 0.05 for alpha may be attributed to Sir Ronald Fisher's initial suggestion (Fisher, 1958, p.16;Bangdiwala, 2016) 2 , even though Fisher later argued the use of a default across all research scenarios for making an automatic conclusion is "absurdly academic" (Fisher, 1973, p. 44-45; 3 .…”

Section: On a Conditional Justification Of The Error Rates In Classic...mentioning

confidence: 99%

On the Credibility of Statistical Inference Within the Neyman-Pearson Hypothesis Testing Framework

Wong,

Mulder

2023

Preprint

View full text Add to dashboard Cite

Neyman-Pearson based Null Hypothesis Significance Testing (NHST) belongs to one of the most commonly used methods in psychology. Carefully specifying and justifying the alpha (α) and beta (β) levels, which reflect the Type I and Type II error probabilities respectively, are crucial steps. As argued by Neyman and Pearson, these choices should depend on the context of the research at hand. However, clear guidelines for doing this are missing. As a result, the conventional level of α = .05 is often used as a default option and the consideration of β is often ignored. Importantly, the use of the conventional α-level is argued to be one of the leading causes of replication crisis. To address this problem, the goal of the current paper is therefore to propose an intuitive conditional justification of the error rates to guide applied researchers to choose a specific set of alpha and beta for each unique research scenario and to improve the credibility of research findings in psychology and related fields. More specifically, the selection of alpha and beta is based on (i) the proportion of true null hypotheses in the field of research and (ii) on the conditional probabilities of (in)correct conclusions given the decisions of the test. A Shiny app has been developed to help researchers to compute the new error rates. The presented method aims to encourage researchers to move away from conventional levels and, instead, strive to rationalize and customize their choices of alpha and beta for each research scenario. This shift aims to facilitate the attainment of credible and trustworthy conclusions, moving beyond automated decision-making based on conventional error rates.

show abstract

The combination of reporting bias and underpowered study designs has substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: A meta-analysis

McKay¹,

Bacelar²,

Parma³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Enhanced expectancies and autonomy-support through self-controlled practice conditions form the motivation pillar of OPTIMAL theory (Wulf & Lewthwaite, 2016). The influence of these practice variables on motor learning was recently evaluated in two separate meta-analyses. Both meta-analyses found that the published literature suggested a moderate and significant benefit on motor learning; however, evidence for reporting bias was found in both literatures. Although multiple bias-corrected estimates were reported in the self-controlled meta-analysis, there was no principled way to prefer one over the other. In the enhanced expectancies meta-analysis, the trim-and-fill-technique failed to correct the estimated effects. Here, we addressed these limitations by reanalyzing the data from both meta-analyses using robust Bayesian meta-analysis methods. Our reanalysis revealed that reporting bias substantially exaggerated the benefits of these practice variables in the original meta-analyses. The true effects are instead small, uncertain, and potentially null. We also found the estimated average statistical power among all studies from the original meta-analyses was 6% (95% confidence interval [5%, 13%]). These results provide compelling and converging evidence that strongly suggests the available literature is insufficient to support the motivation pillar of OPTIMAL theory. Our results highlight the need for adequately powered experimental designs if motor learning scientists want to make evidence-based recommendations.

show abstract

Low prevalence of a priori power analyses in motor behavior research

Cited by 3 publications

References 0 publications

The combination of reporting bias and underpowered study designs has substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: a meta-analysis

The combination of reporting bias and underpowered study designs has substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: a meta-analysis

On the Credibility of Statistical Inference Within the Neyman-Pearson Hypothesis Testing Framework

The combination of reporting bias and underpowered study designs has substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: A meta-analysis

Contact Info

Product

Resources

About