Dealing with substantial heterogeneity in Cochrane reviews. Cross-sectional study

Abstract

BackgroundDealing with heterogeneity in meta-analyses is often tricky, and there is only limited advice for authors on what to do. We investigated how authors addressed different degrees of heterogeneity, in particular whether they used a fixed effect model, which assumes that all the included studies are estimating the same true effect, or a random effects model where this is not assumed.MethodsWe sampled randomly 60 Cochrane reviews from 2008, which presented a result in its first meta-analysis with substantial heterogeneity (I2 greater than 50%, i.e. more than 50% of the variation is due to heterogeneity rather than chance). We extracted information on choice of statistical model, how the authors had handled the heterogeneity, and assessed the methodological quality of the reviews in relation to this.ResultsThe distribution of heterogeneity was rather uniform in the whole I2 interval, 50-100%. A fixed effect model was used in 33 reviews (55%), but there was no correlation between I2 and choice of model (P = 0.79). We considered that 20 reviews (33%), 16 of which had used a fixed effect model, had major problems. The most common problems were: use of a fixed effect model and lack of rationale for choice of that model, lack of comment on even severe heterogeneity and of reservations and explanations of its likely causes. The problematic reviews had significantly fewer included trials than other reviews (4.3 vs. 8.0, P = 0.024). The problems became less pronounced with time, as those reviews that were most recently updated more often used a random effects model.ConclusionOne-third of Cochrane reviews with substantial heterogeneity had major problems in relation to their handling of heterogeneity. More attention is needed to this issue, as the problems we identified can be essential for the conclusions of the reviews.

Alcohol and mortality after moderate to severe traumatic brain injury: a meta-analysis of observational studies

“…Heterogeneity was dealt with by sensitivity analysis, omitting one study at a time to detect interstudy bias, after which these were excluded. 39 The pooled estimates (pooled odds ratio) were obtained using the MantelHaenszel fixed-effect method. 50 BAC was dichotomized to negative and positive for the analyses.…”

“…Dealing with study heterogeneity is a complicated matter with no definitive solution. 39 We chose to deal with heterogeneity by using a fixed-effect method and omitting one study at a time to identify studies causing heterogeneity so these could be excluded for sensitivity analysis.…”

Human Milk Feeding as a Protective Factor for Retinopathy of Prematurity: A Meta-analysis

“…21 Furman et al, 2003 22 Heller et al, 2007 18 Maayan-Metzger et al, 2012 23 Manzoni et al, 2013 17 12 8,12 A randomeffects model for conducting metaanalysis has been suggested as a reasonable way of addressing heterogeneity in pooled studies, and in such cases narratively explaining the reasons for heterogeneity has been proposed. 38 We justify using our meta-analysis approach on the basis of these methodological suggestions.…”

Untitled

“…The random effects model assumes that the effects in the different models follow a distribution, and confidence intervals combine between-study and within-study variances. 12 The outcomes of intrao perative variables, donor surgical complications and graft-related urologic complications, delayed graft function, and graft loss in the recipients were summarized using means and 95% confidence intervals.…”