A guide to systematic review and meta-analysis of prediction model performance

Abstract

Validation of prediction models is highly recommended and increasingly common in the literature. A systematic review of validation studies is therefore helpful, with meta-analysis needed to summarise the predictive performance of the model being validated across different settings and populations. This article provides guidance for researchers systematically reviewing and meta-analysing the existing evidence on a specific prediction model, discusses good practice when quantitatively summarising the predictive performance of the model across studies, and provides recommendations for interpreting meta-analysis estimates of model performance. We present key steps of the meta-analysis and illustrate each step in an example review, by summarising the discrimination and calibration performance of the EuroSCORE for predicting operative mortality in patients undergoing coronary artery bypass grafting. Systematic reviews and meta-analysis are an importantif not the most important-source of information for evidence based medicine. 1 Traditionally, they aim to summarise the results of publications or reports of primary treatment studies and (more recently) of primary diagnostic test accuracy studies. Compared to therapeutic intervention and diagnostic test accuracy studies, there is limited guidance on the conduct of systematic reviews and meta-analysis of primary prognosis studies.A common aim of primary prognostic studies concerns the development of so-called prognostic prediction models or indices. These models estimate the individualised probability or risk that a certain condition will occur in the future by combining information from multiple prognostic factors from an individual. Unfortunately, there is often conflicting evidence about the predictive performance of developed prognostic prediction models. For this reason, there is a growing demand for evidence synthesis of (external validation) studies assessing a model's performance in new individuals. 2 A similar issue relates to diagnostic prediction models, where the validation performance of a model for predicting the risk of a disease being already present is of interest across multiple studies.Previous guidance papers regarding methods for systematic reviews of predictive modelling studies have addressed the searching, 3-5 design, 2 data extraction, and critical appraisal 6 7 of primary studies. In this paper, we provide further guidance for systematic review and for meta-analysis of such models. Systematically reviewing the predictive performance of one or more prediction models is crucial to examine a model's predictive ability across different study populations, settings, or locations, [8][9][10][11] and to evaluate the need for further adjustments or improvements of a model.Although systematic reviews of prediction modelling studies are increasingly common, 12-17 researchers often refrain from undertaking a quantitative synthesis or meta-analysis of the predictive performance of a specific model. Potential reasons for this pitfall are concerns about the qu...

A guide to systematic review and meta-analysis of prognostic factor studies

“…Each objective requires specific methods and tools for conducting a systematic review and meta-analysis. Two recent articles provided a guide to undertaking reviews and meta-analysis of prognostic (prediction) models 67. In this article, we focus on prognostic factors.…”

When and how to use data from randomised trials to develop or validate prognostic models

“…Secondly, knowledge of the operationalisation of predictor or outcome measurements provides insight into how well a model can perform in practice, and can inform the assessment of the risk of bias when reviewing a prognostic model study 26. In addition, information on how and when variables were collected and recorded might have predictive value.…”

External validation, update and development of prediction models for pre-eclampsia using an Individual Participant Data (IPD) meta-analysis: the International Prediction of Pregnancy Complication Network (IPPIC pre-eclampsia) protocol

“…This will be done on the logit scale for the Cstatistic, as it is unlikely to be normally distributed on the original scale. Within-imputation SEs can be obtained on these transformed scale by applying the delta-method and using the formulae given by Debray et al [48].…”

“…Therefore the logit scale will be used to pool across imputations (as this is also the scale that will be used later in the meta-analysis) [52]. The SE for logit(Cstatistic) can be calculated from the C-statistic and SE for the C-statistic using the following formula [48]:…”

“…Random-effects will be used rather than fixed-effect meta-analysis because it seems reasonable that the performance of a model may differ across populations due to case-mix [48,54]. Random-effects metaanalysis will also allow us to quantify any heterogeneity in performance across studies and predict model performance in other similar settings using approximate 95% prediction intervals [55].…”