On the moments of Cochran's Q statistic under the null hypothesis, with application to the meta‐analysis of risk difference

Abstract

W. G. Cochran's Q statistic was introduced in 1937 to test for equality of means under heteroscedasticity. Today, the use of Q is widespread in tests for homogeneity of effects in meta-analysis, but often these effects (such as risk differences and odds ratios) are not normally distributed. It is common to assume that Q follows a chi-square distribution, but it has long been known that this asymptotic distribution for Q is not accurate for moderate sample sizes. In this paper, the effect and weight for an individual study may depend on two parameters: the effect and a nuisance parameter. We present expansions for the first two moments of Q without any normality assumptions. Our expansions will have wide applicability in testing for homogeneity in meta-analysis. As an important example, we present a homogeneity test when the effects are the differences of risks between treatment and control arms of the several studies-a test which is substantially more accurate than that currently used. In this situation, we approximate the distribution of Q with a gamma distribution. We provide the results of simulations to verify the accuracy of our proposal and an example of a meta-analysis of medical data. Copyright © 2012 John Wiley & Sons, Ltd.

Confidence intervals for the between-study variance in random-effects meta-analysis using generalised heterogeneity statistics: should we use unequal tails?

“…This means that the random-effects model, and so the methods used here, can be quite a crude approximation when applied to real data. Kulinskaya and colleagues [34, 35] show that the distribution of quadratic forms in meta-analysis, when applied to real data, differ from their theoretical distributions under the random-effects model. We regard this as a serious problem only when the studies are small, although this can quite often be the case in application.…”

Practical challenges of I2 as a measure of heterogeneity

“…Third, that average value of Q has a different form for each choice of the measure of effect. Kulinskaya et al investigated the standardized mean difference and the risk difference, and Kulinskaya and Dollinger investigated the odds ratio. In all these cases, the formula for the average value involves constants that must be estimated from the data in each meta‐analysis.…”

Approximate confidence intervals for moment‐based estimators of the between‐study variance in random effects meta‐analysis

“…However, Kulinskaya et al . (, ) show that results concerning quadratic forms in meta‐analysis that rely on the assumptions of the random effects model can be poor approximations when applied to real datasets, and all the methods that follow are subject to these issues. Methods that take into account the fact that the within‐study variances are estimated have been proposed (Böhning et al, ; Malzahn et al, ), but at present, our methodology does not attempt this.…”