A note on the graphical presentation of prediction intervals in random-effects meta-analyses

Abstract

BackgroundMeta-analysis is used to combine the results of several related studies. Two different models are generally applied: the fixed-effect (FE) and random-effects (RE) models. Although the two approaches estimate different parameters (that is, the true effect versus the expected value of the distribution of true effects) in practice, the graphical presentation of results is the same for both models. This means that in forest plots of RE meta-analyses, no estimate of the between-study variation is usually given graphically, even though it provides important information about the heterogeneity between the study effect sizes.FindingsIn addition to the point estimate of the between-study variation, a prediction interval (PI) can be used to determine the degree of heterogeneity, as it provides a region in which about 95% of the true study effects are expected to be found. To distinguish between the confidence interval (CI) for the average effect and the PI, it may also be helpful to include the latter interval in forest plots. We propose a new graphical presentation of the PI; in our method, the summary statistics in forest plots of RE meta-analyses include an additional row, ‘95% prediction interval’, and the PI itself is presented in the form of a rectangle below the usual diamond illustrating the estimated average effect and its CI. We then compare this new graphical presentation of PIs with previous proposals by other authors. The way the PI is presented in forest plots is crucial. In previous proposals, the distinction between the CI and the PI has not been made clear, as both intervals have been illustrated either by a diamond or by extra lines added to the diamond, which may result in misinterpretation.ConclusionsTo distinguish graphically between the results of an FE and those of an RE meta-analysis, it is helpful to extend forest plots of the latter approach by including the PI. Clear presentation of the PI is necessary to avoid confusion with the CI of the average effect estimate.

Type 2 diabetes and risk of cancer

“…Unlike 95% confidence intervals, 95% prediction intervals do not provide information about the average effect size or its statistical significance. 5 A 95% prediction interval that includes the null value of 1 should not be interpreted as a null association overall but merely as an indication that the estimate obtained in the next study might not always be significantly different from the null. In general, substantial heterogeneity could lead to a wide prediction interval, in which case it would be important to identify the sources of heterogeneity.…”

Network Meta-Analysis of Adjuvant Chemotherapy following Resection of Colorectal Liver Metastases

“…A random-effects model was used to calculate 95% prediction intervals (PIs), which were reported to assess further statistical heterogeneity. These 95% PIs demonstrate the effect variance among studies and can predict a more conservative summary treatment effect of a future similar trial [25,26].…”

“…The node splitting method was used to calculate the inconsistency of the model that separated direct from indirect evidence; the agreement between the two was evaluated and reported with Bayesian values [25,26].…”

A simple method to estimate prediction intervals and predictive distributions: Summarizing meta‐analyses beyond means and confidence intervals

“…There was substantial heterogeneity ( I 2 was 92.5%), and a random‐effects model should be used. The mean ( –0.715; exp[−0.715] = 0.49, the overall RR reported by Colditz et al) and variance ( 0.318) of the heterogeneity were automatically calculated (Figure ) based on the DL method, which made no assumption of the underlying distribution. The DL method also gave the estimated standard error of the mean, so the confidence interval of was readily available (Figure ).…”

“…Recently, researchers have argued that a summary estimate (such as a mean with a confidence interval) provides an incomplete summary of the underlying distribution . The prediction interval, which shows the range of true effects in future studies, has been advocated to be regularly presented in meta‐analyses . Most commonly, prediction intervals are estimated assuming that the underlying heterogeneity follows a normal distribution …”