Accounting for treatment use when validating a prognostic model: a simulation study

Pajouheshnia, Romin; Peelen, Linda M.; Moons, Karel G.M.; Reitsma, Johannes B.; Groenwold, Rolf H H

doi:10.1186/s12874-017-0375-8

Cited by 36 publications

(48 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This may have resulted into the prevention of sPTB and thus an underestimation of model discrimination and calibration 40. One of the selected studies, Alleman et al.…”

Section: Discussionmentioning

confidence: 99%

Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation

Meertens

Montfort

Scheepers

et al. 2018

Acta Obstet Gynecol Scand

View full text Add to dashboard Cite

IntroductionPrediction models may contribute to personalized risk‐based management of women at high risk of spontaneous preterm delivery. Although prediction models are published frequently, often with promising results, external validation generally is lacking. We performed a systematic review of prediction models for the risk of spontaneous preterm birth based on routine clinical parameters. Additionally, we externally validated and evaluated the clinical potential of the models.Material and methodsPrediction models based on routinely collected maternal parameters obtainable during first 16 weeks of gestation were eligible for selection. Risk of bias was assessed according to the CHARMS guidelines. We validated the selected models in a Dutch multicenter prospective cohort study comprising 2614 unselected pregnant women. Information on predictors was obtained by a web‐based questionnaire. Predictive performance of the models was quantified by the area under the receiver operating characteristic curve (AUC) and calibration plots for the outcomes spontaneous preterm birth <37 weeks and <34 weeks of gestation. Clinical value was evaluated by means of decision curve analysis and calculating classification accuracy for different risk thresholds.ResultsFour studies describing five prediction models fulfilled the eligibility criteria. Risk of bias assessment revealed a moderate to high risk of bias in three studies. The AUC of the models ranged from 0.54 to 0.67 and from 0.56 to 0.70 for the outcomes spontaneous preterm birth <37 weeks and <34 weeks of gestation, respectively. A subanalysis showed that the models discriminated poorly (AUC 0.51–0.56) for nulliparous women. Although we recalibrated the models, two models retained evidence of overfitting. The decision curve analysis showed low clinical benefit for the best performing models.ConclusionsThis review revealed several reporting and methodological shortcomings of published prediction models for spontaneous preterm birth. Our external validation study indicated that none of the models had the ability to predict spontaneous preterm birth adequately in our population. Further improvement of prediction models, using recent knowledge about both model development and potential risk factors, is necessary to provide an added value in personalized risk assessment of spontaneous preterm birth.

show abstract

“…This may have resulted into the prevention of sPTB and thus an underestimation of model discrimination and calibration 40. One of the selected studies, Alleman et al.…”

Section: Discussionmentioning

confidence: 99%

Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation

Meertens

Montfort

Scheepers

et al. 2018

Acta Obstet Gynecol Scand

View full text Add to dashboard Cite

show abstract

“…It follows that models developed using data from individuals who received guided treatments will provide biased underestimates of (untreated) risks in future individuals, if treatment use is ignored [8]. In validation studies, models will incorrectly appear to overestimate risk if applied in individuals that receive the specific guided treatment [8,11].…”

Section: Guided Treatmentsmentioning

confidence: 99%

“…prognosis with or without treatment), as well as the types of treatments (guided or background) used in a data set or study, are key factors that determine how treatments may impact on prognostic model development or validation. For further details on the challenges of treatment use and how to account for them in prognostic model development and validation, see [8] and [11], respectively, and further guidance can be found in Table 1 (see below).…”

Section: Examplesmentioning

confidence: 99%

See 1 more Smart Citation

Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies

et al. 2017

Self Cite

View full text Add to dashboard Cite

Background: Ignoring treatments in prognostic model development or validation can affect the accuracy and transportability of models. We aim to quantify the extent to which the effects of treatment have been addressed in existing prognostic model research and provide recommendations for the handling and reporting of treatment use in future studies. Methods: We first describe how and when the use of treatments by individuals in a prognostic study can influence the development or validation of a prognostic model. We subsequently conducted a systematic review of the handling and reporting of treatment use in prognostic model studies in cardiovascular medicine. Data on treatment use (e.g. medications, surgeries, lifestyle interventions), the timing of their use, and the handling of such treatment use in the analyses were extracted and summarised. Results: Three hundred two articles were included in the review. Treatment use was not mentioned in 91 (30%) articles. One hundred forty-six (48%) reported specific information about treatment use in their studies; 78 (26%) provided information about multiple treatments. Three articles (1%) reported changes in medication use ("treatment drop-in") during follow-up. Seventy-nine articles (26%) excluded treated individuals from their analysis, 80 articles (26%) modelled treatment as an outcome, and of the 155 articles that developed a model, 86 (55%) modelled treatment use, almost exclusively at baseline, as a predictor. Conclusions: The use of treatments has been partly considered by the majority of CVD prognostic model studies. Detailed accounts including, for example, information on treatment drop-in were rare. Where relevant, the use of treatments should be considered in the analysis of prognostic model studies, particularly when a prognostic model is designed to guide the use of certain treatments and these treatments have been used by the study participants. Future prognostic model studies should clearly report the use of treatments by study participants and consider the potential impact of treatment use on the study findings.

show abstract

“…For this purpose, prognostic models must predict risks for patients in the absence of a certain treatment—which can prove challenging in non-RCT data, because of the non-random use of treatments by patients,28 and because advanced statistical methods might be needed to correctly account for this 2930. In the case of RCT data, the effect of treatment use can be solved by simply developing or validating the prognostic model in the control trial arm (control treatment, untreated, or placebo treated) or by including the randomised treatment as a predictor in the model, along with terms for any other treatment-predictor interactions (model development only) 24.…”

Section: Opportunities Arising From Rct Data Usementioning

confidence: 99%

When and how to use data from randomised trials to develop or validate prognostic models

Pajouheshnia

Groenwold

Peelen

et al. 2019

BMJ

Self Cite

View full text Add to dashboard Cite

Prediction models have become an integral part of clinical practice, providing information for patients and clinicians and providing support for their shared decision making. The development and validation of prognostic prediction models requires substantial volumes of high quality information on relevant predictors and patient health outcomes. Primary data collection dedicated to prognostic model (development or validation) research could come with substantial time and costs and can be seen as a waste of resources if suitable data are already available. Randomised clinical trials are a source of high quality clinical data with a largely untapped potential for use in further research. This article addresses when and how data from a randomised clinical trial can be used additionally for prognostic model research, and provides guidance for researchers with access to trial data to evaluate the suitability of their data for the development and validation of prognostic prediction models.

show abstract

Accounting for treatment use when validating a prognostic model: a simulation study

Cited by 36 publications

References 27 publications

Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation

Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation

Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies

When and how to use data from randomised trials to develop or validate prognostic models

Contact Info

Product

Resources

About