Hyperparameters and tuning strategies for random forest

Abstract

The random forest (RF) algorithm has several hyperparameters that have to be set by the user, for example, the number of observations drawn randomly for each tree and whether they are drawn with or without replacement, the number of variables drawn randomly for each split, the splitting rule, the minimum number of samples that a node must contain, and the number of trees. In this paper, we first provide a literature review on the parameters' influence on the prediction performance and on variable importance measures. It is well known that in most cases RF works reasonably well with the default values of the hyperparameters specified in software packages. Nevertheless, tuning the hyperparameters can improve the performance of RF. In the second part of this paper, after a presenting brief overview of tuning strategies, we demonstrate the application of one of the most established tuning strategies, model‐based optimization (MBO). To make it easier to use, we provide the tuneRanger R package that tunes RF with MBO automatically. In a benchmark study on several datasets, we compare the prediction performance and runtime of tuneRanger with other tuning implementations in R and RF with default hyperparameters. This article is categorized under: Algorithmic Development > Biological Data Mining Algorithmic Development > Statistics Algorithmic Development > Hierarchies and Trees Technologies > Machine Learning

The physiological effects of non-invasive brain stimulation fundamentally differ across the human cortex

“…We validated the modular decomposition using permutation testing on three levels: (i) the individual unthresholded FC-matrix and (ii) the co-classification matrix of each participant, as well as (iii) the co-classification matrix at the group level (Dwyer et al 2014 leakage of information from the test to the training data. We evaluated the performance of the classifier using (i) a nested-cross validation (leaving out the two observations corresponding to one subject for testing) and (ii) an inner validation approach for the hyperparameter optimization of the random forest-classifier (using a sequential model-based optimization implemented by the Scikit-Optimize library; skopt, https://github.com/scikit-optimize/scikit-optimize), iteratively tuning the following parameters following the recommendations by Probst et al (Probst, Wright and Boulesteix 2019): maximum depth of the tree, number of features, minimum number of samples and minimum number of samples required to be at a leaf node . We statistically validated the observed accuracy using permutation testing ( p < 0.05, 5000 iterations) randomizing the class labels.…”

Character displacement within the breeding area questions reinforcement inFicedulaflycatchers

“…We used the open-source R software environment for statistical computing and graphics (version 3.5.0) under an integrated development environment for R -RStudio 295 (RStudio Desktop version 1.1.447) to analyse data assembled on step V. For regression tasks we used the ranger package (version 0.10.1) as an implementation of the random forests (Wright and Ziegler 2017). To obtain the most accurate predictions, the random forest parameters need to be optimised (Probst et al 2018). To configure the parameters of the random forest, we used the tuneRanger (version 0.3) package (Probst et al 2018) 300 which allows model-based optimization for tuning strategy and the three parameters min.node.size, sample.fraction and mtry tuning at once.…”

“…To obtain the most accurate predictions, the random forest parameters need to be optimised (Probst et al 2018). To configure the parameters of the random forest, we used the tuneRanger (version 0.3) package (Probst et al 2018) 300 which allows model-based optimization for tuning strategy and the three parameters min.node.size, sample.fraction and mtry tuning at once. Out-of-bag predictions were used for evaluation.…”

The Effect of Interfirm Financial Transactions on the Credit Risk of Small and Medium-Sized Enterprises

“…Compared with logistic regression, random forests requires tuning for some of the parameters. We used the function OOBCurve in the homonymous R package (Probst and Boulesteix, ) to tune the number of trees, whereas we use the latest implementations in the tuneRanger R package (Probst et al ., ) to tune the number of variables randomly sampled as candidates at each split, the minimal size of terminal nodes and the fraction of observations to sample (function tuneRanger). In all cases, we choose the area under the curve (AUC) as the performance criterion for tuning.…”