Assessment of a Clinical Trial–Derived Survival Model in Patients With Metastatic Castration-Resistant Prostate Cancer

Abstract

IMPORTANCE Randomized clinical trials (RCTs) are considered the criterion standard for clinical evidence. Despite their many benefits, RCTs have limitations, such as costliness, that may reduce the generalizability of their findings among diverse populations and routine care settings. OBJECTIVE To assess the performance of an RCT-derived prognostic model that predicts survival among patients with metastatic castration-resistant prostate cancer (CRPC) when the model is applied to real-world data from electronic health records (EHRs). DESIGN, SETTING, AND PARTICIPANTS The RCT-trained model and patient data from the RCTs were obtained from the Dialogue for Reverse Engineering Assessments and Methods (DREAM) challenge for prostate cancer, which occurred from March 16 to July 27, 2015. This challenge included 4 phase 3 clinical trials of patients with metastatic CRPC. Real-world data were obtained from the EHRs of a tertiary care academic medical center that includes a comprehensive cancer center. In this study, the DREAM challenge RCT-trained model was applied to real-world data from January 1, 2008, to December 31, 2019; the model was then retrained using EHR data with optimized feature selection. Patients with metastatic CRPC were divided into RCT and EHR cohorts based on data source. Data were analyzed from March 23, 2018, to October 22, 2020. EXPOSURES Patients who received treatment for metastatic CRPC. MAIN OUTCOMES AND MEASURES The primary outcome was the performance of an RCT-derived prognostic model that predicts survival among patients with metastatic CRPC when the model is applied to real-world data. Model performance was compared using 10-fold cross-validation according to time-dependent integrated area under the curve (iAUC) statistics. RESULTS Among 2113 participants with metastatic CRPC, 1600 participants were included in the RCT cohort, and 513 participants were included in the EHR cohort. The RCT cohort comprised a larger proportion of White participants (1390 patients [86.9%] vs 337 patients [65.7%]) and a smaller proportion of Hispanic participants (14 patients [0.9%] vs 42 patients [8.2%]), Asian participants (41 patients [2.6%] vs 88 patients [17.2%]), and participants older than 75 years (388 patients [24.3%] vs 191 patients [37.2%]) compared with the EHR cohort. Participants in the RCT cohort also had fewer comorbidities (mean [SD], 1.6 [1.8] comorbidities vs 2.5 [2.6] comorbidities, respectively) compared with those in the EHR cohort. Of the 101 variables used in the RCT-derived model, 10 were not available in the EHR data set, 3 of which were among the top 10 features in the DREAM challenge RCT model. The best-performing EHR-trained model included only 25 of the 101 variables included in the RCT-trained model. The performance of the RCT-trained and EHR-trained models was adequate in the EHR cohort (mean [SD] iAUC, 0.722 [0.118] and 0.762 [0.106], respectively); model optimization was associated with improved performance of the best-performing EHR model (mean [SD] iAUC, (continued) Key Points ...

Machine Learning Applied to Electronic Health Records: Identification of Chemotherapy Patients at High Risk for Preventable Emergency Department Visits and Hospital Admissions

“…Given the prior evidence of inequitable outcomes on the basis of demographic subgroup and insurer, we felt it was important to provide race, ethnicity, and insurance data to our models to study their respective effects. [32][33][34][35][36] Our results further indicate that Black race and Medicaid payor are predictive of increased risk for ACU. In particular, our findings that patients with cancer with Medicaid insurance have higher risk scores for ACU are aligned with previous findings and suggest that Medicaid is correlated with poor patient outcomes.…”

Building Predictive Models for Clinical Care—Where to Build and What to Predict?

“…However, performance of this top-performing EHR-derived model was modest (integrated area under the receiver operating characteristic curve, 0.596) when evaluated on a small test data set (10% of the 513 patients had been withheld from model development). 1 Despite the modest size of the metastatic CRPC sample, the study 1 supports the feasibility of using EHR data to apply and evaluate predictive models. We offer 2 suggestions for extending this work.…”

“…Initial application of the RCT-derived model resulted in a modest reduction in model fit compared with its performance in the RCT test data set, in part because 17 of the 101 model variables from the RCT-derived model, including several key predictors, were unavailable in the EHR. 1 When remaining EHR-derived variables were used to retrain and optimize the model, a 25-variable model achieved the same integrated area under the receiver operating characteristic curve (0.792) as the original model had achieved in the RCT cohort. However, performance of this top-performing EHR-derived model was modest (integrated area under the receiver operating characteristic curve, 0.596) when evaluated on a small test data set (10% of the 513 patients had been withheld from model development).…”

“…Coquet et al1 discuss advantages and disadvantages of RCT data for this purpose. Advantages include confidence that patients actually have the specific disease under study by virtue of strict + Related article Author affiliations and article information are listed at the end of this article.…”

Prognostic Association between Common Laboratory Tests and Overall Survival in Elderly Men with De Novo Metastatic Castration Sensitive Prostate Cancer: A Population-Based Study in Canada

“…The Gravis model was based on a very small number of patients with relatively low discriminatory ability and the Akamatsu model was based on a retrospective review of a small number of patients with only Asian ethnicity. Consequently, these models should be validated in larger, more diverse, real-world populations [ 21 ]. Additionally, the prognostic values of other easily accessible laboratory parameters (e.g., markers of inflammation, testosterone levels, etc.)…”