Firth's logistic regression with rare events: accurate effect estimates and predictions?

Puhr, Rainer; Heinze, Georg; Nold, Mariana; Lusa, Lara; Geroldinger, Angelika

doi:10.1002/sim.7273

Cited by 248 publications

(251 citation statements)

References 16 publications

Supporting

Mentioning

243

Contrasting

Unclassified

Order By: Relevance

“…As overall performance is composed of discrimination and calibration, the improvement in calibration improves the overall performance. Our results indeed showed that the overall performance was slightly better for penalized than for unpenalized MLR, which is in agreement with earlier simulation studies on binary logistic regression …”

Section: Discussionsupporting

confidence: 92%

Sample size considerations and predictive performance of multinomial logistic prediction models

Jong

Eijkemans

Calster

et al. 2019

Statistics in Medicine

View full text Add to dashboard Cite

Multinomial Logistic Regression (MLR) has been advocated for developing clinical prediction models that distinguish between three or more unordered outcomes. We present a full‐factorial simulation study to examine the predictive performance of MLR models in relation to the relative size of outcome categories, number of predictors and the number of events per variable. It is shown that MLR estimated by Maximum Likelihood yields overfitted prediction models in small to medium sized data. In most cases, the calibration and overall predictive performance of the multinomial prediction model is improved by using penalized MLR. Our simulation study also highlights the importance of events per variable in the multinomial context as well as the total sample size. As expected, our study demonstrates the need for optimism correction of the predictive performance measures when developing the multinomial logistic prediction model. We recommend the use of penalized MLR when prediction models are developed in small data sets or in medium sized data sets with a small total sample size (ie, when the sizes of the outcome categories are balanced). Finally, we present a case study in which we illustrate the development and validation of penalized and unpenalized multinomial prediction models for predicting malignancy of ovarian cancer.

show abstract

Section: Discussionsupporting

confidence: 92%

Sample size considerations and predictive performance of multinomial logistic prediction models

Jong

Eijkemans

Calster

et al. 2019

Statistics in Medicine

View full text Add to dashboard Cite

show abstract

“…Puhr et al . () proposed the FLAC approach to overcome the bias problem in average predicted probabilities of the Firth estimator. The method involves the following steps.…”

Section: Models and Predictive Accuracy Measuresmentioning

confidence: 99%

“…This is in line with the submission in Puhr et al . (), who opined that the Firth method introduces bias in predicted probabilities. Ridge in contrast does not perform well at EPV=2 when the AUROC is used to evaluate discrimination.…”

Section: Simulation Based On the Taiwan Credit Card Data Setmentioning

confidence: 99%

“…Since statistical models for rare events are rarely used in this framework, we compared Firth's () method, a modification of Firth's logistic regression called Firth's logistic regression with added covariate (FLAC, proposed by Puhr et al . ()) and logistic regression penalized by a log‐ F prior (LF22, by Greenland and Mansournia ()) with methods such as linear discriminant analysis (LDA), ridge regression (Ridge), the GEV model and logistic regression. In particular, we ‘balanced’ the data by using random oversampling examples (ROSE, by Menardi and Torelli () and Lunardon et al .…”

Section: Introductionmentioning

confidence: 99%

“…The aim of this paper is therefore to evaluate the performance of statistical approaches for rare events and EPV-requirements for accurate predictions in credit scoring. Since statistical models for rare events are rarely used in this framework, we compared Firth's (1993) method, a modification of Firth's logistic regression called Firth's logistic regression with added covariate (FLAC, proposed by Puhr et al (2017)) and logistic regression penalized by a log-F prior (LF22, by Greenland and Mansournia (2015)) with methods such as linear discriminant analysis (LDA), ridge regression (Ridge), the GEV model and logistic regression. In particular, we 'balanced' the data by using random oversampling examples (ROSE, by Menardi and Torelli (2014) and Lunardon et al (2014)) and the synthetic minority oversampling technique SMOTE (by Chawla et al (2002)) methods for various EPV-values and develop prediction models by using logistic regression.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Prediction of Default Probability by using Statistical Models for rare Events

Ogundimu

2019

Journal of the Royal Statistical Society Series A: Statistics in Society

View full text Add to dashboard Cite

Summary Prediction models in credit scoring usually involve the use of data sets with highly imbalanced distributions of the event of interest (default). Logistic regression, which is widely used to estimate the probability of default, PD, often suffers from the problem of separation when the event of interest is rare and consequently poor predictive performance of the minority class in small samples. A common solution is to discard majority class examples, to duplicate minority class examples or to use a combination of both to balance the data. These methods may overfit data. It is unclear how penalized regression models such as Firth's estimator, which reduces bias and mean‐square error relative to classical logistic regression, performs in modelling PD. We review some methods for class imbalanced data and compare them in a simulation study using the Taiwan credit card data. We emphasize the effect of events per variable for developing an accurate model—an often neglected concept in PD‐modelling. The data balancing techniques that are considered are the random oversampling examples and synthetic minority oversampling technique methods. The results indicate that the synthetic minority oversampling technique improved predictive accuracy of PD regardless of sample size. Among the penalized regression models that are analysed, the log‐F prior and ridge regression methods are preferred.

show abstract

Factors Associated With Suicide Ideation in US Army Soldiers During Deployment in Afghanistan

et al. 2020

View full text Add to dashboard Cite

IMPORTANCE Understanding suicide ideation (SI) during combat deployment can inform prevention and treatment during and after deployment. OBJECTIVE To examine associations of sociodemographic characteristics, lifetime and past-year stressors, and mental disorders with 30-day SI among a representative sample of US Army soldiers deployed in Afghanistan. DESIGN, SETTING, AND PARTICIPANTS In this survey study, soldiers deployed to Afghanistan completed self-administered questionnaires in July 2012. The sample was weighted to represent all 87 032 soldiers serving in Afghanistan. Prevalence of lifetime, past-year, and 30-day SI and mental disorders was determined. Logistic regression analyses examined risk factors associated with SI. Data analyses for this study were conducted between August 2018 and August 2019. MAIN OUTCOMES AND MEASURES Suicide ideation, lifetime and 12-month stressors, and mental disorders were assessed with questionnaires. Administrative records identified sociodemographic characteristics and suicide attempts. RESULTS A total of 3957 soldiers (3473 [weighted 87.5%] male; 2135 [weighted 52.6%] aged Յ29 years) completed self-administered questionnaires during their deployment in Afghanistan. Lifetime, past-year, and 30-day SI prevalence estimates were 11.7%, 3.0%, and 1.9%, respectively. Among soldiers with SI, 44.2% had major depressive disorder (MDD) and 19.3% had posttraumatic stress disorder in the past 30-day period. A series of analyses of the 23 grouped variables potentially associated with SI resulted in a final model of sex; race/ethnicity; lifetime noncombat trauma; past 12-month relationship problems, legal problems, and death or illness of a friend or family member; and MDD. In this final multivariable model, white race/ethnicity (odds ratio [OR], 3.1 [95% CI, 1.8-5.1]), lifetime noncombat trauma (OR, 2.1 [95% CI, 1.1-4.0]), and MDD (past 30 days: OR, 31.8 [95% CI, 15.0-67.7]; before past 30 days: OR, 4.9 [95% CI, 2.5-9.6]) were associated with SI. Among the 85 soldiers with past 30-day SI, from survey administration through 12 months after returning from deployment, 6% (5 participants) had a documented suicide attempt vs 0.14% (6 participants) of the 3872 soldiers without SI. CONCLUSIONS AND RELEVANCE This study suggests that major depressive disorder and noncombat trauma are important factors in identifying SI risk during combat deployment.

show abstract

Firth's logistic regression with rare events: accurate effect estimates and predictions?

Cited by 248 publications

References 16 publications

Sample size considerations and predictive performance of multinomial logistic prediction models

Sample size considerations and predictive performance of multinomial logistic prediction models

Prediction of Default Probability by using Statistical Models for rare Events

Factors Associated With Suicide Ideation in US Army Soldiers During Deployment in Afghanistan

Contact Info

Product

Resources

About