Random forest versus logistic regression: a large-scale benchmark experiment

Abstract

Background and goalThe Random Forest (RF) algorithm for regression and classification has considerably gained popularity since its introduction in 2001. Meanwhile, it has grown to a standard classification approach competing with logistic regression in many innovation-friendly scientific fields.ResultsIn this context, we present a large scale benchmarking experiment based on 243 real datasets comparing the prediction performance of the original version of RF with default parameters and LR as binary classification tools. Most importantly, the design of our benchmark experiment is inspired from clinical trial methodology, thus avoiding common pitfalls and major sources of biases.ConclusionRF performed better than LR according to the considered accuracy measured in approximately 69% of the datasets. The mean difference between RF and LR was 0.029 (95%-CI =[0.022,0.038]) for the accuracy, 0.041 (95%-CI =[0.031,0.053]) for the Area Under the Curve, and − 0.027 (95%-CI =[−0.034,−0.021]) for the Brier score, all measures thus suggesting a significantly better performance of RF. As a side-result of our benchmarking experiment, we observed that the results were noticeably dependent on the inclusion criteria used to select the example datasets, thus emphasizing the importance of clear statements regarding this dataset selection process. We also stress that neutral studies similar to ours, based on a high number of datasets and carefully designed, will be necessary in the future to evaluate further variants, implementations or parameters of random forests which may yield improved accuracy compared to the original version with default values.Electronic supplementary materialThe online version of this article (10.1186/s12859-018-2264-5) contains supplementary material, which is available to authorized users.

Novel application of approaches to predicting medication adherence using medical claims data

“…Although advanced analytic methods and traditional regression models have comparable discrimination, model performance is often influenced by both the size of the cohort under study and the number of events per variable (EPV) . Evidence suggests that logistic regression models perform better (in terms of accuracy, parsimony and/or discrimination) in smaller datasets with approximately 20‐50 EPV, while random forest models perform well with larger sample sizes and achieve sufficient stability when EPV exceeds 200 . In addition, the number and type of variables selected as predictors affect model performance.…”

“…In addition, the number and type of variables selected as predictors affect model performance. Previous research has demonstrated that random forest models achieve high performance not only as more variables are selected, but also when a large number of continuous variables are used as predictors . Given these considerations, data composition, quality, and completeness should be of the utmost importance when selecting or merging clinical data sources for prediction modeling.…”

“…[30][31][32][33] Evidence suggests that logistic regression models perform better (in terms of accuracy, parsimony and/or discrimination) in smaller datasets with approximately 20-50 EPV, while random forest models perform well with larger sample sizes and achieve sufficient stability when EPV exceeds 200. 30,31,34,35 In addition, the number and type of variables selected as predictors affect model performance. Previous research has demonstrated that random forest models achieve high performance not only as more variables are selected, but also when a large number of continuous variables are used as predictors.…”

A Machine‐Learning Approach to Identify a Prognostic Cytokine Signature That Is Associated With Nivolumab Clearance in Patients With Advanced Melanoma

“…Random forest, a tree-based ensemble algorithm, is a widely used machine-learning algorithm [23][24][25] and has proven to be a robust machine-learning method that is popular in many applications. [26][27][28][29] Pharmacokinetic (nivolumab clearance) and pharmacodynamic (inflammatory cytokine panel) data from 468 patients with melanoma treated with nivolumab in two phase III studies (CheckMate 066 and 067) were used as training data sets. Summary statistics of cytokines used in the development of the machine-learning model are provided in Table S1.…”

The Effect of Interfirm Financial Transactions on the Credit Risk of Small and Medium-Sized Enterprises

“…In the next sections, we use these chosen variables within a credit risk model and assess their effect on the prediction of default. We do so both by using a standard logistic regression approach, which transparently quantifies the effect of each individual predictor on credit risk and which is traditionally used in the credit risk industry, and a random-forest approach (Breiman, 2001), which is a more flexible machine learning method based on an ensemble of decision trees which has been found to perform well in many classification tasks and under various benchmark studies (Couronné et al, 2018). Table 4 shows the results of a logistic regression analysis.…”