Evaluation of multiple prediction models: A novel view on model selection and performance assessment

Westphal, Max; Brannath, Werner

doi:10.1177/0962280219854487

Cited by 11 publications

(39 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To arrive at a suitable selection of prediction models for the evaluation study, more work is necessary when sensitivity and specificity are assessed simultaneously. The within 1 SE rule 37 can be adopted to the co‐primary endpoint scenario by basing it on the weighted accuracy

wAcc = w Se + (1 - w) Sp

w \in (0, 1)

instead of the overall classification accuracy. We will investigate the performance of this approach with the weight

w = {(\frac{1 - {Se}_{0}}{1 - {Sp}_{0}} + 1)}^{- 1}

in Section 5.…”

Section: Model Selection Prior To the Evaluation Studymentioning

confidence: 99%

A multiple testing framework for diagnostic accuracy studies with co‐primary endpoints

Westphal

Zapf

Brannath

2022

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

Major advances have been made regarding the utilization of machine learning techniques for disease diagnosis and prognosis based on complex and high‐dimensional data. Despite all justified enthusiasm, overoptimistic assessments of predictive performance are still common in this area. However, predictive models and medical devices based on such models should undergo a throughout evaluation before being implemented into clinical practice. In this work, we propose a multiple testing framework for (comparative) phase III diagnostic accuracy studies with sensitivity and specificity as co‐primary endpoints. Our approach challenges the frequent recommendation to strictly separate model selection and evaluation, that is, to only assess a single diagnostic model in the evaluation study. We show that our parametric simultaneous test procedure asymptotically allows strong control of the family‐wise error rate. A multiplicity correction is also available for point and interval estimates. Moreover, we demonstrate in an extensive simulation study that our multiple testing strategy on average leads to a better final diagnostic model and increased statistical power. To plan such studies, we propose a Bayesian approach to determine the optimal number of models to evaluate simultaneously. For this purpose, our algorithm optimizes the expected final model performance given previous (hold‐out) data from the model development phase. We conclude that an assessment of multiple promising diagnostic models in the same evaluation study has several advantages when suitable adjustments for multiple comparisons are employed.

show abstract

wAcc = w Se + (1 - w) Sp

w \in (0, 1)

instead of the overall classification accuracy. We will investigate the performance of this approach with the weight

w = {(\frac{1 - {Se}_{0}}{1 - {Sp}_{0}} + 1)}^{- 1}

in Section 5.…”

Section: Model Selection Prior To the Evaluation Studymentioning

confidence: 99%

A multiple testing framework for diagnostic accuracy studies with co‐primary endpoints

Westphal

Zapf

Brannath

2022

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

show abstract

“…We will investigate the finite sample FWER in realistic and least-favorable settings in section 4. As described by Westphal and Brannath (2019a), we extend ϕ to a multiple test for the actually relevant hypothesis system…”

Section: Study Goalmentioning

confidence: 99%

“…Firstly, the extended test retains (asymptotic) FWER control as only non-rejections are added. Secondly, we can compare different model selection strategies because the extended multiple test always operates on H = H M and not only on H S (Westphal & Brannath, 2019a).…”

Section: Study Goalmentioning

confidence: 99%

“…can thus be calculated as the column means of these similarity matrices (Westphal & Brannath, 2019a, 2019b. Moreover, we can estimate the covariances Σ Se = cov( Se) and Σ Sp = cov( Sp) as the sample covariance matrices of the similarity matrices divided by factors of n 1 and n 0 , respectively.…”

Section: Parameter Estimationmentioning

confidence: 99%

“…Our goal is to apply the so-called maxT-approach to hypothesis system (2) (Hothorn et al, 2008). This simultaneous test procedure has previously been utilized for the assessment of the overall classification accuracy of several binary classifiers (Westphal & Brannath, 2019a, 2019b. This multiple test enables us to reduce the need for multiplicity adjustment in case several similar models are evaluated.…”

Section: Statistical Inferencementioning

confidence: 99%

See 2 more Smart Citations

A multiple testing framework for diagnostic accuracy studies with co-primary endpoints

Westphal¹,

Zapf²,

Brannath³

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Major advances have been made regarding the utilization of artificial intelligence in health care. In particular, deep learning approaches have been successfully applied for automated and assisted disease diagnosis and prognosis based on complex and high-dimensional data. However, despite all justified enthusiasm, overoptimistic assessments of predictive performance are still common. Automated medical testing devices based on machine-learned prediction models should thus undergo a throughout evaluation before being implemented into clinical practice. In this work, we propose a multiple testing framework for (comparative) phase III diagnostic accuracy studies with sensitivity and specificity as co-primary endpoints. Our approach challenges the frequent recommendation to strictly separate model selection and evaluation, i.e. to only assess a single diagnostic model in the evaluation study. We show that our parametric simultaneous test procedure asymptotically allows strong control of the family-wise error rate. Moreover, we demonstrate in extensive simulation studies that our multiple testing strategy on average leads to a better final diagnostic model and increased statistical power. To plan such studies, we propose a Bayesian approach to determine the optimal number of models to evaluate. For this purpose, our algorithm optimizes the expected final model performance given previous (hold-out) data from the model development phase. We conclude that an assessment of multiple promising diagnostic models in the same evaluation study has several advantages when suitable adjustments for multiple comparisons are conducted.

show abstract

Machine learning allows for large-scale habitat prediction of a wide-ranging carnivore across diverse ecoregions

O’Malley,

Elbroch,

Zeller

et al. 2024

Landsc Ecol

View full text Add to dashboard Cite

Context Resource selection functions are powerful tools for predicting habitat selection of animals. Recently, machine-learning methods such as random forest have gained popularity for predicting habitat selection due to their flexibility and strong predictive performance. Objectives We tested two methods for predicting continental-scale, second-order habitat selection of a wide-ranging large carnivore, the mountain lion (Puma concolor), to support continent-wide conservation management, including estimating abundance, and to predict habitat suitability for recolonizing or reintroduced animals. Methods We compared a generalized linear model (GLM) and a random forest model using GPS location data from 476 individuals across 20 study sites in the western USA and Canada and remotely-sensed landscape data. We internally validated models and examined their ability to correctly classify used and available points by calculating area under the receiver operating characteristics (AUC). We performed leave-one-out (LOO) out-of-sample tests of predictive strength on both models. Results Both models suggested that mountain lions select for steeper slopes, areas closer to water, and with higher normalized difference vegetation index (NDVI), and against variables associated with human impact. The random forest model (AUC = 0.94) demonstrated that mountain lion habitat can be accurately predicted at continental scales, outperforming the traditional GLM model (AUC = 0.68). Our LOO validation provided similar results (x̄ = 0.93 for the random forest and x̄ = 0.65 for the GLM). Conclusions We found that the added flexibility of the random forest model provided deeper insights into how individual covariates impacted habitat selection across diverse ecosystems. Our LOO analyses suggested that our model can predict mountain lion habitat selection in unoccupied areas or where local data are unavailable. Our model thus provides a tool to support discussions and analyses relevant to continent-wide mountain lion conservation and management including estimating metapopulation abundance.

show abstract

Evaluation of multiple prediction models: A novel view on model selection and performance assessment

Cited by 11 publications

References 36 publications

A multiple testing framework for diagnostic accuracy studies with co‐primary endpoints

A multiple testing framework for diagnostic accuracy studies with co‐primary endpoints

A multiple testing framework for diagnostic accuracy studies with co-primary endpoints

Machine learning allows for large-scale habitat prediction of a wide-ranging carnivore across diverse ecoregions

Contact Info

Product

Resources

About