A comparison of 20 heterogeneity variance estimators in statistical synthesis of results from studies: a simulation study

Meta-analyses often include only a small number of studies (≤5). Estimating between-study heterogeneity is difficult in this situation. An inaccurate estimation of heterogeneity can result in biased effect estimates and too narrow confidence intervals. The beta-binominal model has shown good statistical properties for meta-analysis of sparse data. We compare the beta-binominal model with different inverse variance random (eg, DerSimonian-Laird, modified Hartung-Knapp, and Paule-Mandel) and fixed effects methods (Mantel-Haenszel and Peto) in a simulation study. The underlying true parameters were obtained from empirical data of actually performed meta-analyses to best mirror real-life situations. We show that valid methods for meta-analysis of a small number of studies are available. In fixed effects situations, the Mantel-Haenszel and Peto methods performed best. In random effects situations, the beta-binominal model performed best for meta-analysis of few studies considering the balance between coverage probability and power. We recommended the beta-binominal model for practical application. If very strong evidence is needed, using the Paule-Mandel heterogeneity variance estimator combined with modified Hartung-Knapp confidence intervals might be useful to confirm the results. Notable most inverse variance random effects models showed unsatisfactory statistical properties also if more studies (10-50) were included in the meta-analysis.

Section: Discussionmentioning

confidence: 62%

A comparison of methods for meta‐analysis of a small number of studies with binary outcomes

Mathes

Kuß

2018

“…Any τ 2 estimator can be used when computing a WTz CI. 50,113,114 This method often has coverage probability considerably below nominal 0.95 level 14,15,45,[115][116][117][118][119][120][121] when k is small and/or τ 2 is large. 13,45,52,79,118,119,[122][123][124][125] Brockwell and Gordon 13 stated that the greatest source of error in the method is the use of a normal approximation for μ̂R E .…”

Section: Resultsmentioning

confidence: 99%

Methods to calculate uncertainty in the estimated overall effect size from a random‐effects meta‐analysis

Veroniki

Jackson

Bender

et al. 2018

154

210

Meta-analyses are an important tool within systematic reviews to estimate the overall effect size and its confidence interval for an outcome of interest. If heterogeneity between the results of the relevant studies is anticipated, then a random-effects model is often preferred for analysis. In this model, a prediction interval for the true effect in a new study also provides additional useful information. However, the DerSimonian and Laird method-frequently used as the default method for meta-analyses with random effects-has been long challenged due to its unfavorable statistical properties. Several alternative methods have been proposed that may have better statistical properties in specific scenarios. In this paper, we aim to provide a comprehensive overview of available methods for calculating point estimates, confidence intervals, and prediction intervals for the overall effect size under the random-effects model. We indicate whether some methods are preferable than others by considering the results of comparative simulation and real-life data studies.

“…And

θ_{j} = θ + ε_{j},

ε_{j} ~ N ((),, 0, τ^{2}),

where τ 2 is the between‐study variance. Numerous estimators exist for τ 2 , with the one most commonly used proposed by DerSimonian and Laird:

{\overset{truê}{τ}}_{DL}^{2} = \frac{Q_{\overset{truê}{w}} - ((), k - 1)}{\sum_{j = 1}^{k} {\overset{truê}{w}}_{j} - \sum_{j = 1}^{k} {\overset{truê}{w}}_{j}^{2} / \sum_{j = 1}^{k} {\overset{truê}{w}}_{j}},

Q_{\overset{truê}{w}} = \sum_{j = 1}^{k} {\overset{truê}{w}}_{j} {({\overset{γ}{true}}_{1 j} - {\overset{γ}{true}}_{1})}^{2},

where

{\overset{truê}{w}}_{j} = 1 / σ_{j}^{2}

, k is the number of studies, and

{\overset{truê}{γ}}_{1} = \frac{\sum_{j = 1}^{k} {\overset{truê}{w}}_{j} {\overset{´}{Y}}_{j}}{\sum_{j = 1}^{k} {\overset{truê}{w}}_{j}}

is the overall exposure effect estimate under a fixed‐effect approach with

{\overset{´}{Y}}_{j} = θ + e_{j}

.…”

Section: Methodsmentioning

confidence: 99%

“…It is known that the correct specification of the 1‐stage model is critical, and guides and software to aid researchers in this are available . Fewer modelling options are available for two‐stage analysis, and they are mainly around the choice of the second step model, fixed effect, or one of the numerous random‐effects options . A well‐developed implementation of a two‐stage analysis is available in Stata, with the less flexible ipdmeta command, which can only analyse a continuous outcome, available in R software .…”

Section: Introductionmentioning

confidence: 99%

“…[9][10][11] Fewer modelling options are available for two-stage analysis, and they are mainly around the choice of the second step model, fixed effect, or one of the numerous random-effects options. 12,13 A well-developed implementation of a two-stage analysis is available in Stata, 14 with the less flexible ipdmeta command, which can only analyse a continuous outcome, available in R software. 15 One-stage analyses rely on widely used mixedeffects models and are considered more flexible, 16 but they are more challenging in both conducting and communicating the findings, especially regarding visualisation with the hallmark forest plot, although software solutions are available.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A comparison of one‐stage vs two‐stage individual patient data meta‐analysis methods: A simulation study

Kontopantelis

2018

BackgroundIndividual patient data (IPD) meta‐analysis allows for the exploration of heterogeneity and can identify subgroups that most benefit from an intervention (or exposure), much more successfully than meta‐analysis of aggregate data. One‐stage or two‐stage IPD meta‐analysis is possible, with the former using mixed‐effects regression models and the latter obtaining study estimates through simpler regression models before aggregating using standard meta‐analysis methodology. However, a comprehensive comparison of the two methods, in practice, is lacking.MethodsWe generated 1000 datasets for each of many simulation scenarios covering different IPD sizes and different between‐study variance (heterogeneity) assumptions at various levels (intercept and exposure). Numerous simulation settings of different assumptions were also used, while we evaluated performance both on main effects and interaction effects. Performance was assessed on mean bias, mean error, coverage, and power.ResultsFully specified one‐stage models (random study intercept or fixed study‐specific intercept; random exposure effect; and fixed study‐specific effects for covariate) were the best performers overall, especially when investigating interactions. For main effects, performance was almost identical across models unless intercept heterogeneity was present, in which case the fully specified one‐stage and the two‐stage models performed better. For interaction effects, differences across models were greater with the two‐stage model consistently outperformed by the two fully specified one‐stage models.ConclusionsA fully specified one‐stage model should be preferred (accounting for potential exposure, intercept, and, possibly, interaction heterogeneity), especially when investigating interactions. If non‐convergence is encountered with a random study intercept, the fixed study‐specific intercept one‐stage model should be used instead.