Nested logistic regression models and ΔAUC applications: Change-point analysis

Abstract

The area under the receiver operating characteristic curve (AUC) is one of the most popular measures for evaluating the performance of a predictive model. In nested models, the change in AUC (ΔAUC) can be a discriminatory measure of whether the newly added predictors provide significant improvement in terms of predictive accuracy. Recently, several authors have shown rigorously that ΔAUC can be degenerate and its asymptotic distribution is no longer normal when the reduced model is true, but it could be the distribution of a linear combination of some [Formula: see text] random variables [ 1 , 2 ]. Hence, the normality assumption and existing variance estimate cannot be applied directly for developing a statistical test under the nested models. In this paper, we first provide a brief review on the use of ΔAUC for comparing nested logistic models and the difficulty of retrieving the reference distribution behind. Then, we present a special case of the nested logistic regression models that the newly added predictor to the reduced model contains a change-point in its effects. A new test statistic based on ΔAUC is proposed in this setting. A simple resampling scheme is proposed to approximate the critical values for the test statistic. The inference of the change-point parameter is done via m-out-of- n bootstrap. Large-scale simulation is conducted to evaluate the finite-sample performance of the ΔAUC test for the change-point model. The proposed method is applied to two real-life datasets for illustration.

Performance of Radiological and Biochemical Biomarkers in Predicting Radio-Symptomatic Knee Osteoarthritis Progression

“…As described earlier, the reference model involved the use of baseline clinical covariates (CLIN: age, sex, and BMI), radiological readings (KL and JSNM), and TBT descriptors, whereas the proposed model involved, in addition, the use of longitudinal variations in TBT (∆TBT) parameters. All models were developed using nested logistic regression [ 5 , 7 , 10 , 21 ] to evaluate their prediction performance based on the TBT parameters of the complete tibial subchondral bone structure (Models 1–5), or those of the medial tibial plateau (TBTM) (Model 6), lateral tibial plateau (TBTL) (Model 7), or central tibial plateau (TBTC) (Model 8). Figure 1 shows the location of these regions.…”

“…As described earlier, the reference model invo ates (CLIN: age, sex, and BMI), radiological reading whereas the proposed model involved, in addition TBT (∆TBT) parameters. All models were develo [5,7,10,21] to evaluate their prediction performanc complete tibial subchondral bone structure (Models eau (TBTM) (Model 6), lateral tibial plateau (TBT (TBTC) (Model 8). Figure 1 shows the location of the evaluated using a 10-fold cross-validation, repeated…”

Modeling and estimating a threshold effect: An application to improving cardiac surgery practices

“…Therefore, it may be preferable to incorporate this knowledge into modeling to help better estimate the threshold. Other relevant work on modeling and/or estimating change-points or thresholds include, for example, Fong et al, 21 Chen et al, 27 Elder and Fong, 28 Elliott and Shope, 29 Fong, 30 Lee, 31 Pastor-Barriuso et al, 32 and Tapsoba et al 33 Using the nadir DO2 example as a case study, this paper discusses and compares various models and methods that can be used to estimate a threshold, and demonstrates the potential bias of popular existing methods. In Section 2, we describe two existing models, namely, the sudden-jump model and the broken-stick model, and common estimation methods under each model.…”

“…Therefore, it may be preferable to incorporate this knowledge into modeling to help better estimate the threshold. Other relevant work on modeling and/or estimating change-points or thresholds include, for example, Fong et al, 21 Chen et al, 27 Elder and Fong, 28 Elliott and Shope, 29 Fong, 30 Lee, 31 Pastor-Barriuso et al, 32 and Tapsoba et al 33…”