Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study

Ban, Jong-Wook; Emparanza, José Ignacio; Urreta, Iratxe; Burls, Amanda

doi:10.1371/journal.pone.0145779

Cited by 15 publications

(14 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This confirms that case-control studies should not be used to study certain aspects of model calibration. Further, we found that the total OE ratio was influenced by the method of outcome assessment, in agreement with previous studies that showed that higher diagnostic odds ratios were found in studies with differential outcome verification 13. We also expected to find lower OE ratios when the validation population differed from the development population (eg, in terms of case-mix) 14.…”

Section: Discussionsupporting

confidence: 90%

“…We could not systematically confirm this across all reviews, likely caused by heterogeneity between systematic reviews as indicated by the wide confidence intervals. Finally, we could not fully confirm the association between sample size and model performance that was previously found,13 although we found similar trends in part of the reviews.…”

Section: Discussioncontrasting

confidence: 87%

“…For diagnostic prediction models, evidence suggests estimates of performance may be biased in studies with certain study characteristics. One study found a higher diagnostic odds ratio in case-control studies, studies with differential outcome verification (ie, using different outcome assessments across study individuals), and with low sample size 13. To date, no meta-epidemiological study has been performed investigating the possible impact of study characteristics on measures of the predictive performance of a prognostic model on external validation, which is commonly quantified in terms of discrimination and calibration 14.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

et al. 2019

View full text Add to dashboard Cite

ObjectivesTo empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models.DesignMeta-epidemiological study.Data sources and study selectionOn 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model.Data extraction and analysesStudy design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance.ResultsWe included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63).ConclusionsVariation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.

show abstract

Section: Discussionsupporting

confidence: 90%

Section: Discussioncontrasting

confidence: 87%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

et al. 2019

View full text Add to dashboard Cite

show abstract

“…This study has some important limitations. 33 One of the most important limitations is the performance of the study in the same cardiac surgery center. This could lead to the unknowing duplication of the same errors that may contribute to the development of delirium, and which will not occur in another center.…”

mentioning

confidence: 99%

Cardiac Delirium Index for Predicting the Occurrence of Postoperative Delirium in Adult Patients After Coronary Artery Bypass Grafting

Lechowicz

Szylińska

Listewnik

et al. 2021

CIA

View full text Add to dashboard Cite

Background: Postoperative delirium (POD) is a serious complication of cardiac surgery. It is an acute neuropsychiatric syndrome. The aim of this study was to analyze the CARDEL Index, composed of advancing age, preoperative glycated hemoglobin and the platelet-to-WBC ratio (PWR) previously described and calculated, using a different patient database, to assess its usefulness as a marker for predicting postoperative delirium after coronary artery bypass grafting (CABG). Methods: A retrospective analysis of 1098 patients who underwent, isolated CABG procedures between 2017 and 2019 was performed. Results: Within the study group, 164/1098 (14.93%) patients were diagnosed with delirium. Preoperative inflammatory parameters were elevated in patients with delirium: White Blood Cell count (p=0.003), Neutrophil count (p=0.016) and C-reactive protein (p<0.001). A decrease in preoperative PWR was shown in patients with delirium (p=0.008). Delirious patients spent more time mechanically ventilated (p<0.001) and had longer hospitalization times (p=0.002). Mortality at 1 year was significantly higher in patients with POD (p<0.001). The CARDEL Index in this study group for POD detection has the largest area under the curve (AUC) of 0.664 (p<0.001) and a cutoff value of 8.08. Conclusion: CARDEL Index may be treated as a potentially valuable tool for delirium prediction in patients after CABG.

show abstract

“…This may help to understand under what circumstances the model performance remains adequate, and when the model might require further improvements. As mentioned earlier, the discrimination and calibration of a prediction model can be affected by differences in the design 38 and in populations across the validation studies, for example, owing to changes in case mix variation or baseline risk. 8 22 In general, sources of heterogeneity can be explored by performing a meta-regression analysis where the dependent variable is the (transformed) estimate of the model performance measure.…”

Section: Investigating Heterogeneity Across Studiesmentioning

confidence: 99%

A guide to systematic review and meta-analysis of prediction model performance

Debray

Damen

Snell

et al. 2017

BMJ

390

531

View full text Add to dashboard Cite

Validation of prediction models is highly recommended and increasingly common in the literature. A systematic review of validation studies is therefore helpful, with meta-analysis needed to summarise the predictive performance of the model being validated across different settings and populations. This article provides guidance for researchers systematically reviewing and meta-analysing the existing evidence on a specific prediction model, discusses good practice when quantitatively summarising the predictive performance of the model across studies, and provides recommendations for interpreting meta-analysis estimates of model performance. We present key steps of the meta-analysis and illustrate each step in an example review, by summarising the discrimination and calibration performance of the EuroSCORE for predicting operative mortality in patients undergoing coronary artery bypass grafting. Systematic reviews and meta-analysis are an importantif not the most important-source of information for evidence based medicine. 1 Traditionally, they aim to summarise the results of publications or reports of primary treatment studies and (more recently) of primary diagnostic test accuracy studies. Compared to therapeutic intervention and diagnostic test accuracy studies, there is limited guidance on the conduct of systematic reviews and meta-analysis of primary prognosis studies.A common aim of primary prognostic studies concerns the development of so-called prognostic prediction models or indices. These models estimate the individualised probability or risk that a certain condition will occur in the future by combining information from multiple prognostic factors from an individual. Unfortunately, there is often conflicting evidence about the predictive performance of developed prognostic prediction models. For this reason, there is a growing demand for evidence synthesis of (external validation) studies assessing a model's performance in new individuals. 2 A similar issue relates to diagnostic prediction models, where the validation performance of a model for predicting the risk of a disease being already present is of interest across multiple studies.Previous guidance papers regarding methods for systematic reviews of predictive modelling studies have addressed the searching, 3-5 design, 2 data extraction, and critical appraisal 6 7 of primary studies. In this paper, we provide further guidance for systematic review and for meta-analysis of such models. Systematically reviewing the predictive performance of one or more prediction models is crucial to examine a model's predictive ability across different study populations, settings, or locations, [8][9][10][11] and to evaluate the need for further adjustments or improvements of a model.Although systematic reviews of prediction modelling studies are increasingly common, 12-17 researchers often refrain from undertaking a quantitative synthesis or meta-analysis of the predictive performance of a specific model. Potential reasons for this pitfall are concerns about the qu...

show abstract

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study

Cited by 15 publications

References 58 publications

Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

Cardiac Delirium Index for Predicting the Occurrence of Postoperative Delirium in Adult Patients After Coronary Artery Bypass Grafting

A guide to systematic review and meta-analysis of prediction model performance

Contact Info

Product

Resources

About