Estimating the Area under the ROC Curve When Transporting a Prediction Model to a Target Population

Abstract

We propose methods for estimating the area under the receiver operating characteristic (ROC) curve (AUC) of a prediction model in a target population that differs from the source population that provided the data used for original model development. If covariates that are associated with model performance, as measured by the AUC, have a different distribution in the source and target populations, then AUC estimators that only use data from the source population will not reflect model performance in the target population. Here, we provide identification results for the AUC in the target population when outcome and covariate data are available from the sample of the source population, but only covariate data are available from the sample of the target population. In this setting, we propose three estimators for the AUC in the target population and show that they are consistent and asymptotically normal. We evaluate the finite‐sample performance of the estimators using simulations and use them to estimate the AUC in a nationally representative target population from the National Health and Nutrition Examination Survey for a lung cancer risk prediction model developed using source population data from the National Lung Screening Trial.

Differential gene expression analysis pipelines and bioinformatic tools for the identification of specific biomarkers: A review

“…Being independent of disease prevalence and specific decision criteria, the AUC provides a reliable indication of test performance and facilitates comparison between different diagnostic tests [154] . However, it is important to emphasize that evaluating the effectiveness of diagnostic tests requires an appropriate study design, which should include a representative study population, with clearly defined inclusion and exclusion criteria to ensure that the patient’s pool reflects the diversity of the general population [155] . The inclusion in the study population of a wide range of cases and controls is crucial for accurate and reliable results and should be accompanied with appropriate randomization procedures to minimize bias and confounders [156] .…”

Illusory generalizability of clinical prediction models

“…The second key risk is poor model transportability. Models may lack external validity due to patients, providers, or implementation characteristics varying across trials ( 12 ).…”