Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors

Abstract

Multivariable regression models are powerful tools that are used frequently in studies of clinical outcomes. These models can use a mixture of categorical and continuous variables and can handle partially observed (censored) responses. However, uncritical application of modelling techniques can result in models that poorly fit the dataset at hand, or, even more likely, inaccurately predict outcomes on new subjects. One must know how to measure qualities of a model's fit in order to avoid poorly fitted or overfitted models. Measurement of predictive accuracy can be difficult for survival time data in the presence of censoring. We discuss an easily interpretable index of predictive discrimination as well as methods for assessing calibration of predicted survival probabilities. Both types of predictive accuracy should be unbiasedly validated using bootstrapping or cross-validation, before using predictions in a new data series. We discuss some of the hazards of poorly fitted and overfitted regression models and present one modelling strategy that avoids many of the problems discussed. The methods described are applicable to all regression models, but are particularly needed for binary, ordinal, and time-to-event outcomes. Methods are illustrated with a survival analysis in prostate cancer using Cox regression.

Pancreatic endocrine tumors: improved TNM staging and histopathological grading permit a clinically efficient prognostic stratification of patients

“…Univariate and multivariate analyses were performed using the Cox regression model to evaluate significant 10-year mortality predictors. Concordance (c) index, 20 defined as the proportion of all usable patient pairs in which predictions and outcomes are concordant, was calculated for different Cox's models. The c index estimates the probability of concordance between predicted and observed survival.…”

Multi-institutional Validation of the CAPRA-S Score to Predict Disease Recurrence and Mortality After Radical Prostatectomy

“…The discriminative ability of CAPRA-S to predict metastasis and PCSM was assessed by the c-index [18]. In these analyses, patients who did not have a lymph node dissection were treated as having negative nodes.…”

Self‐Reported Smoking, Urine Cotinine, and Risk of Cardiovascular Disease: Findings From the PREVEND (Prevention of Renal and Vascular End‐Stage Disease) Prospective Cohort Study

“…To assess whether adding information on urine cotinine assessed smoking exposure to conventional cardiovascular risk factors35 is associated with an improvement in the prediction of CVD risk, we calculated measures of discrimination for censored time‐to‐event data (Harrell's C‐index36) and reclassification. To investigate the change in C‐index, we added smoking status to a model on the basis of traditional risk factors included in the Framingham CVD Risk Score (ie, age, sex, systolic blood pressure, total cholesterol, and high‐density lipoprotein cholesterol) 37.…”