A guide for calculating study-level statistical power for meta-analyses

Abstract

Meta-analysis is a popular approach in the psychological sciences for synthesizing data across studies. However, the credibility of meta-analysis outcomes depends on the evidential value of studies included in the body of evidence used for data synthesis. One important consideration for determining a study’s evidential value is the statistical power of the study’s design and statistical test combination for detecting hypothetical effect sizes of interest. Studies with a design/test combination that cannot reliably detect a wide range of effect sizes are more susceptible to questionable research practices and exaggerated effect sizes. Therefore, determining the statistical power for design/test combinations for studies included in meta-analyses can help researchers make decisions regarding confidence in the body of evidence. As the one true population effect size is unknown when hypothesis testing, a better approach is to determine statistical power for a range of hypothetical effect sizes. This tutorial introduces the metameta R package and web app, which facilitates the straightforward calculation and visualization of study-level statistical power in meta-analyses for a range of hypothetical effect sizes. Readers will be shown how to re-analyze data from published meta-analysis and how to integrate the metameta package when reporting novel meta-analyses. A step-by-step companion screencast video tutorial is also provided to assist readers using the R package.

Board diversity and firm innovation: a meta-analysis

“…By applying the statistical power calculator (an R script and excel file) for meta-analysis developed by Quintana and Tiebel (2018)–available at: https://osf.io/5c7uz/–and taking into account the presence of high heterogeneity–as shown by the I 2 statistic: 89.3%–the calculated statistical power of the conducted meta-analysis is 0.96. This is well above the conventional minimum acceptable statistical power for meta-analyses–which has, following Cohen (1992), commonly been set to 0.80–indicating that the utilised sample is “large enough” for the purposes of meta-analysis.…”

Effects of exogenous oxytocin administration on non-social executive function in humans: A preregistered systematic review and meta-analysis protocol

“…Two reviewers (HK and BG) will independently take part in the assessment and discuss to reach consensus upon any disagreement. The statistical power of studies included in the meta-analysis will also be calculated and visualized using the metameta R package (Quintana, 2022).…”

“…However, contrary to initial assumptions that oxytocin enhances prosocial cognition and behavior, oxytocin has also been found under some conditions to increase antisocial emotions such as envy, gloating (Shamay-Tsoory et al, 2009), and aggressiveness (Ne'eman et al, 2016). Moreover, the credibility of early studies has been challenged due to statistical and methodological issues (Leng & Ludwig, 2016;Quintana, 2022;Quintana, 2020;Walum et al, 2016). Meta-analyses have also demonstrated inconsistent effects of oxytocin on emotional stimuli processing (Hedge's g=0.…”

Does the mere presence of a smartphone impact cognitive performance? A meta-analysis of the “brain drain effect”

“…In a second post-hoc, exploratory analysis, we considered the evidential value of the body of literature on the mere presence effect in terms of statistical power. To do so, we used the metameta R package (Quintana, 2022) to estimate the median statistical power for each meta-analysis for a range of true effect sizes as well as the summary effect size obtained in each meta-analysis. Figure 8 depicts a firepower plot summarizing this analysis.…”