Dealing with effect size multiplicity in systematic reviews and meta‐analyses

Abstract

Systematic reviews often encounter primary studies that report multiple effect sizes based on data from the same participants. These have the potential to introduce statistical dependency into the meta-analytic data set. In this paper, we provide a tutorial on dealing with effect size multiplicity within studies in the context of meta-analyses of intervention and association studies, recommending a three-step approach. The first step is to define the research question and consider the extent to which it mainly reflects interest in mean effect sizes (which we term a convergent approach) or an interest in exploring heterogeneity (which we term a divergent approach). A second step is to identify the types of multiplicities that appear in the initial database of effect sizes relevant to the research question, and we propose a categorization scheme to differentiate them. The third step is to select a strategy for dealing with each type of multiplicity. The researcher can choose between a reductionist meta-analytic approach, which is characterized by inclusion of a single effect size per study, and an integrative approach, characterized by inclusion of multiple effect sizes per study. We present an overview of available analysis strategies for dealing with effect size multiplicity within studies and provide recommendations intended to help researchers decide which strategy might be preferable in particular situations. Last, we offer caveats and cautions about addressing the challenges multiplicity poses for systematic reviews and meta-analyses.

Are neuromuscular adaptations present in people with recurrent spinal pain during a period of remission? A protocol for a systematic review

“…Random-effects meta-analysis (in accordance with the Cochrane Back and Neck Group)45 will be considered for a limited number of outcomes in order to reduce the risk of type I error (due to statistical multiplicity)64; therefore, the following outcomes are selected: muscle activity, muscle timing (during a postural perturbation), range of movements (including their variability) and muscle strength. These outcomes were selected considering the feasibility of their assessment in clinical practice and the theoretical framework of motor adaptation to pain 31 38.…”

Comparison of the clinical effectiveness of treatments for aromatase inhibitor-induced arthralgia in breast cancer patients: a protocol for a systematic review and network meta-analysis

“…To obtain SE of a within-study averaged effect, the averaged variance of the effect estimates will be used as the variance of the averaged effect size. 41 Statistical analysis 1. For direct comparisons, if study population, interventions, comparator and measurement time points are homogenous, a standard pairwise meta-analysis will be performed.…”

Spie charts for quantifying treatment effectiveness and safety in multiple outcome network meta-analysis: A proof-of-concept study

“…If there are important dependencies between the outcomes, this should be accounted for at the synthesis stage. There are several approaches available for this and guidance is provided by López-López and colleagues [22] and multi-parameter evidence synthesis methods should also be considered [2]. Nevertheless, if there is a need to avoid double counting the contribution of related outcomes, and we know the correlations between them, the contribution of each outcome to the overall area can be adjusted.…”

“…However, since correlation can be negative, the squares of the pairwise correlations, i.e., the coefficients of determination, i.e., 22 ij ij R   should be used instead. The weight of each outcome can then be proportional or equal to the marginal sums of the squared correlation matrix:…”