Evaluation of Old and New Tests of Heterogeneity in Epidemiologic Meta-Analysis

Abstract

The identification of heterogeneity in effects between studies is a key issue in meta-analyses of observational studies, since it is critical for determining whether it is appropriate to pool the individual results into one summary measure. The result of a hypothesis test is often used as the decision criterion. In this paper, the authors use a large simulation study patterned from the key features of five published epidemiologic meta-analyses to investigate the type I error and statistical power of five previously proposed asymptotic homogeneity tests, a parametric bootstrap version of each of the tests, and tau2-bootstrap, a test proposed by the authors. The results show that the asymptotic DerSimonian and Laird Q statistic and the bootstrap versions of the other tests give the correct type I error under the null hypothesis but that all of the tests considered have low statistical power, especially when the number of studies included in the meta-analysis is small (<20). From the point of view of validity, power, and computational ease, the Q statistic is clearly the best choice. The authors found that the performance of all of the tests considered did not depend appreciably upon the value of the pooled odds ratio, both for size and for power. Because tests for heterogeneity will often be underpowered, random effects models can be used routinely, and heterogeneity can be quantified by means of R(I), the proportion of the total variance of the pooled effect measure due to between-study variance, and CV(B), the between-study coefficient of variation.

A systematic review of socioeconomic differences in food habits in Europe: consumption of cheese and milk

“…A test of heterogeneity was calculated, estimating a Q statistic, which follows a chi-square distribution with degrees of freedom n-1, n being the number of studies included in the analysis. A low P value for this statistic indicates the presence of heterogeneity, which somewhat compromises the validity of the pooled estimates (Takkouche et al, 1999). To pool the individual estimates we used the DerSimonian and Laird's random effect model (DerSimonian & Laird, 1986).…”

Diagnostic value of distal colonic polyps for prediction of advanced proximal neoplasia in an average-risk population undergoing screening colonoscopy

“…44 A brief meta-analyses encompassing this and other relevant studies 18,19 was conducted (results are stated in the discussion section). Meta-analytic summary statistics were calculated based on the methods in Takkouche et al 45 …”

Obesity and renal cell cancer – a quantitative review

“…Between-study variation was modelled as a random effect, and heterogeneity over studies was assessed by the significance of the between-study variance (Berkey et al, 1995;Takkouche et al, 1999). The within-study variance was taken to be the estimated variance of the log relative risks for each study, giving more precise estimates greater weights in the summary measure (Greenland, 1998).…”

“…The regression was weighted by the inverse variance of the log relative risk for each category. The correlation between categories was estimated using a previous method (Greenland and Longnecker, 1992).We used a mixed effects weighted regression model to combine estimates from BMI categories from the individual studies.Between-study variation was modelled as a random effect, and heterogeneity over studies was assessed by the significance of the between-study variance (Berkey et al, 1995;Takkouche et al, 1999). The within-study variance was taken to be the estimated variance of the log relative risks for each study, giving more precise estimates greater weights in the summary measure (Greenland, 1998).…”