A Multi-faceted Mess: A Systematic Review of Statistical Power Analysis in Psychology Journal Articles

Abstract

The over-reliance on the null hypothesis significance testing framework and its accompanying tools has recently been challenged. An example of such a tool is statistical power analysis, which is used to determine how many participants are required to detect a minimally meaningful effect size in the population at a given level of power and Type I error rate. To investigate how power analysis is currently used, this study reviews the reporting of 443 power analyses in high impact psychology journals in 2016 and 2017. We found that many pieces of information required for power analyses are not reported, and effect sizes selected for the procedure are often chosen based on an inappropriate rationale. In light of these findings, we argue that the power analysis procedure forces researchers to compromise in the selection of the different pieces of information required. We offer that researchers should focus on tools beyond traditional power analysis when sample planning, such as precision-based power analysis or simply collecting the largest sample size possible.

Reciprocal associations between executive function and academic achievement: A conceptual replication of Schmitt et al. (2017)

“…The STAR and Pathways samples are existing datasets, thus, a sensitivity power analysis was used to calculate the minimally detectable effect sizes (MDES) given the sample sizes for all statistical analyses (Cribbie, Beribisky, & Alter, 2019;Giner-Sorolla et al, 2019). This provides some context for why we see different rates of significance across the studies for given effect sizes.…”

Power to the People: A Beginner’s Tutorial to Power Analysis using jamovi

“…However, the problem is researchers rarely justify their choice of effect size (Bakker et al, 2020;Cribbie et al, 2019), shedding no insight into the thought process behind their power analysis.…”

“…Power analysis is not the only way to justify your sample size (see Lakens, 2021), but despite increased attention to statistical power, it is still rare to find articles that justified their sample size through power analysis (Chen & Liu, 2019;Guo et al, 2014;Larson & Carbine, 2017). Even for those that do report a power analysis, there are often other problems such as poor justification for the effect size, misunderstanding statistical power, or not making the power analysis reproducible (Bakker et al, 2020;Collins & Watt, 2021;Cribbie et al, 2019). Therefore, we present a tutorial which outlines the key decisions behind power analysis and walk through two of the simplest applications: two independent samples and two dependent samples.…”

“…We have split our tutorial into three parts starting with a recap of the statistical concepts underlying power analysis. There are common misconceptions around useful types of power analysis (Cribbie et al, 2019), so it is important to outline what we are trying to achieve. In part two, we outline the decisions you must make when performing a power analysis, like choosing your alpha, beta, and smallest effect size of interest.…”

Conceptual replication and extension of the relation between the number line estimation task and mathematical competence across seven studies

“…The seven studies used in this manuscript are existing datasets, thus, a sensitivity power analysis was used to calculate the range of minimally detectable effect sizes (MDES) given the sample sizes across the proposed correlations (Cribbie, Beribisky, & Alter, 2019;Giner-Sorolla et al, 2019). Across all proposed moderators, the highest sample size was n = 634 and the smallest sample size was n = 71.…”