Random effects meta‐analysis: Coverage performance of 95% confidence and prediction intervals following REML estimation

Abstract

A random effects meta‐analysis combines the results of several independent studies to summarise the evidence about a particular measure of interest, such as a treatment effect. The approach allows for unexplained between‐study heterogeneity in the true treatment effect by incorporating random study effects about the overall mean. The variance of the mean effect estimate is conventionally calculated by assuming that the between study variance is known; however, it has been demonstrated that this approach may be inappropriate, especially when there are few studies. Alternative methods that aim to account for this uncertainty, such as Hartung–Knapp, Sidik–Jonkman and Kenward–Roger, have been proposed and shown to improve upon the conventional approach in some situations. In this paper, we use a simulation study to examine the performance of several of these methods in terms of the coverage of the 95% confidence and prediction intervals derived from a random effects meta‐analysis estimated using restricted maximum likelihood. We show that, in terms of the confidence intervals, the Hartung–Knapp correction performs well across a wide‐range of scenarios and outperforms other methods when heterogeneity was large and/or study sizes were similar. However, the coverage of the Hartung–Knapp method is slightly too low when the heterogeneity is low (I 2 < 30%) and the study sizes are quite varied. In terms of prediction intervals, the conventional approach is only valid when heterogeneity is large (I 2 > 30%) and study sizes are similar. In other situations, especially when heterogeneity is small and the study sizes are quite varied, the coverage is far too low and could not be consistently improved by either increasing the number of studies, altering the degrees of freedom or using variance inflation methods. Therefore, researchers should be cautious in deriving 95% prediction intervals following a frequentist random‐effects meta‐analysis until a more reliable solution is identified. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Inter‐individual differences in weight change following exercise interventions: a systematic review and meta‐analysis of randomized controlled trials

“…The prediction interval has been described as providing “potentially the most relevant and complete statistical inferences to be drawn from random effects meta‐analyses” . However, we exercise due caution in inferences drawn from the prediction interval given the coverage issues identified in the simulations conducted by Partlett and Riley . These authors reported that the coverage of the interval was particularly poor in cases of low effect heterogeneity and/or markedly variable sample size.…”

“…Such under‐coverage would have no material effect on the derived probability of individual response variance in a future trial being clinically relevant. However, we still consider it prudent to view our prediction interval as approximate, as recommended by Partlett and Riley .…”

A guide to systematic review and meta-analysis of prognostic factor studies

“…The standard deviation of this prognostic factor effect across studies is denoted by τ, and non-zero values suggest there is between-study heterogeneity. Confidence intervals for µ should ideally account for uncertainty in estimated variances (in particular τ),51 and we have found the approach of Hartung-Knapp to be robust for this purpose in most settings 1652. When synthesising prognostic effects on the log scale, the summary results and confidence intervals require back transformation (using the exponential function) to the original scale.…”

Is Clinical Anxiety a Risk or a Protective Factor for Executive Functioning in Youth with ADHD? A Meta-regression Analysis

“…Besides reporting on the confidence intervals (CIs) which quantify the precision of an estimated effect, in the results we also report prediction intervals (PIs) which present the expected range of true study effects in similar, new studies (IntHout et al 2016). It can be particularly informative to inspect PIs when there is high heterogeneity between studies included in a meta-analysis (IntHout et al 2016;Partlett and Riley 2017). Subsequently, we ran separate analyses (QM tests; Viechtbauer 2010) for each EF including one moderator at a time.…”