A Novel Method for Identifying a Parsimonious and Accurate Predictive Model for Multiple Clinical Outcomes

Abstract

Background and Objective: Most methods for developing clinical prognostic models focus on identifying parsimonious and accurate models to predict a single outcome; however, patients and providers often want to predict multiple outcomes simultaneously. As an example, for older adults one is often interested in predicting nursing home admission as well as mortality. We propose and evaluate a novel predictor-selection computing method for multiple outcomes and provide the code for its implementation.Methods: Our proposed algorithm selected the best subset of common predictors based on the minimum average normalized Bayesian Information Criterion (BIC) across outcomes: the Best Average BIC (baBIC) method. We compared the predictive accuracy (Harrell’s C-statistic) and parsimony (number of predictors) of the model obtained using the baBIC method with: 1) a subset of common predictors obtained from the union of optimal models for each outcome (Union method), 2) a subset obtained from the intersection of optimal models for each outcome (Intersection method), and 3) a model with no variable selection (Full method). We used a case-study data from the Health and Retirement Study (HRS) to demonstrate our method and conducted a simulation study to investigate performance.Results: In the case-study data and simulations, the average Harrell’s C-statistics across outcomes of the models obtained with the baBIC and Union methods were comparable. Despite the similar discrimination, the baBIC method produced more parsimonious models than the Union method. In contrast, the models selected with the Intersection method were the most parsimonious, but with worst predictive accuracy, and the opposite was true in the Full method. In the simulations, the baBIC method performed well by identifying many of the predictors selected in the baBIC model of the case-study data most of the time and excluding those not selected in the majority of the simulations.Conclusions: Our method identified a common subset of variables to predict multiple clinical outcomes with superior balance between parsimony and predictive accuracy to current methods.

A comprehensive prognostic tool for older adults: Predicting death, ADL disability, and walking disability simultaneously

“…This comprehensive prognostic tool uses one parsimonious set of variables, and verbally asking about all predictors takes <2 min. Statistically, this state‐of‐the‐art prognostic tool used three recently developed methods that enabled variable selection for three outcomes simultaneously, while also considering the time–cost of each variable in addition to its predictive power 10,11,18 . Our models had good discrimination and calibration with little evidence of overfitting, and the models performed well even when one or two predictors were unavailable.…”

“…We applied statistical methods we previously developed to conduct backward elimination on three prediction models simultaneously and accounting for the time needed to assess a predictor using the time-cost information criterion (TCIC). 10,18 Briefly, the TCIC is an alternative to the BIC used to evaluate model fit, and the TCIC favors the variable that takes less time to assess when two variables have identical predictive abilities. Backward elimination was conducted on all three models simultaneously by applying the TCIC to each model at each backward elimination step and finding the lowest average TCIC across the three models.…”

“…Backward elimination was conducted on all three models simultaneously by applying the TCIC to each model at each backward elimination step and finding the lowest average TCIC across the three models. 18 We used 3 restricted cubic splines with 4 knots for age and BMI. To evaluate the discrimination of the final models, we used an integrated area under the curve (iAUC) using the inverse probability of the censoring weighting method.…”

“…We used Fine and Gray competing risk regression, with death as the competing risk, for the ADL disability prediction model and the walking disability prediction model, using survey weights and robust sandwich variance to account for the complex survey design. We applied statistical methods we previously developed to conduct backward elimination on three prediction models simultaneously and accounting for the time needed to assess a predictor using the time–cost information criterion (TCIC) 10,18 . Briefly, the TCIC is an alternative to the BIC used to evaluate model fit, and the TCIC favors the variable that takes less time to assess when two variables have identical predictive abilities.…”

“…Briefly, the TCIC is an alternative to the BIC used to evaluate model fit, and the TCIC favors the variable that takes less time to assess when two variables have identical predictive abilities. Backward elimination was conducted on all three models simultaneously by applying the TCIC to each model at each backward elimination step and finding the lowest average TCIC across the three models 18 . We used 3 restricted cubic splines with 4 knots for age and BMI.…”