Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study

Abstract

When developing risk prediction models on datasets with limited sample size, shrinkage methods are recommended. Earlier studies showed that shrinkage results in better predictive performance on average. This simulation study aimed to investigate the variability of regression shrinkage on predictive performance for a binary outcome. We compared standard maximum likelihood with the following shrinkage methods: uniform shrinkage (likelihood-based and bootstrap-based), penalized maximum likelihood (ridge) methods, LASSO logistic regression, adaptive LASSO, and Firth’s correction. In the simulation study, we varied the number of predictors and their strength, the correlation between predictors, the event rate of the outcome, and the events per variable. In terms of results, we focused on the calibration slope. The slope indicates whether risk predictions are too extreme (slope < 1) or not extreme enough (slope > 1). The results can be summarized into three main findings. First, shrinkage improved calibration slopes on average. Second, the between-sample variability of calibration slopes was often increased relative to maximum likelihood. In contrast to other shrinkage approaches, Firth’s correction had a small shrinkage effect but showed low variability. Third, the correlation between the estimated shrinkage and the optimal shrinkage to remove overfitting was typically negative, with Firth’s correction as the exception. We conclude that, despite improved performance on average, shrinkage often worked poorly in individual datasets, in particular when it was most needed. The results imply that shrinkage methods do not solve problems associated with small sample size or low number of events per variable.

Development and internal validation of prognostic models to predict negative health outcomes in older patients with multimorbidity and polypharmacy in general practice

“… 76 However, there is an ongoing debate whether it solves such problems, and a recent study has suggested that although shrinkage can result in improved calibration, it may not be superior in terms of reducing overfitting. 77 Furthermore, shrinkage models lead to biassed estimates of the regression coefficients, thus making results more difficult to interpret. Third, in our modelling approach we tested disease-based indicators such as the Charlson Comorbidity Index and CIRS that were developed and validated for other purposes.…”

An advanced prediction model for postoperative complications and early implant failure

“…It is preferable to include shrinkage in the model estimation process using penalized regression (Harrell Jr, 2015). Models based on penalized regression include L1 penalized estimation (least absolute shrinkage and selection operator, lasso), L2 penalized estimation (ridge), and the Firth penalization (Firth, 1993), which is often preferred as the amount of penalization does not have to be estimated from sparse data (Van Calster, van Smeden, De Cock, & Steyerberg, 2020). The mathematics behind these methods are similar.…”

Predicting Prostate Cancer Upgrading of Biopsy Gleason Grade Group at Radical Prostatectomy Using Machine Learning-Assisted Decision-support Models

“…The least absolute shrinkage and selection operator (Lasso) is a popular ML algorithm with outstanding feature selection capability. The LASSO preferentially shrinks some predictor coe cients to zero by penalizing the absolute values of the regression coe cients [15,16]. In this study, the optimized logistic regression coe cients were estimated given a boundary ("L1 Norm") to the sum of absolute standardized regression coe cients [15,16].…”