A computationally fast variable importance test for random forests for high-dimensional data

Abstract

Random forests are a commonly used tool for classification with high-dimensional data as well as for ranking candidate predictors based on the so-called variable importance measures. There are different importance measures for ranking predictor variables, the two most common measures are the Gini importance and the permutation importance. The latter has been found to be more reliable than the Gini importance. It is computed from the change in prediction accuracy when removing any association between the response and a predictor variable, with large changes indicating that the predictor variable is important. A drawback of those variable importance measures is that there is no natural cutoff that can be used to discriminate between important and non-important variables. Several approaches, for example approaches based on hypothesis testing, have been developed for addressing this problem. The existing testing approaches are permutation-based and require the repeated computation of forests. While for low-dimensional settings those permutation-based approaches might be computationally tractable, for high-dimensional settings typically including thousands of candidate predictors, computing time is enormous. A new computationally fast heuristic procedure of a variable importance test is proposed, that is appropriate for high-dimensional data where many variables do not carry any information. The testing approach is based on a modified version of the permutation variable importance measure, which is inspired by cross-validation procedures. The novel testing approach is tested and compared to the permutation-based testing approach of Altmann and colleagues using studies on complex high-dimensional binary classification settings. The new approach controlled the type I error and had at least comparable power at a substantially smaller computation time in our studies. The new variable importance test is implemented in the R package vita.

Divergence in land surface modeling: linking spread to structure

“…The permutation variable importance measure used herein quantifies the loss in skill, the algorithm's ability to predict composite score based on structural indicator variables, by randomly permuting the values of a single predictor variable and comparing that to the unpermuted version. We use this metric to filter structural indicator variables not useful in understanding how structure impacts composite skill, i.e., those with negative values are excluded from further analysis (Janitza et al 2018). Using variable importance scores as a filter and then calculating gain for topmost splitting variables allows us to identify which structural attributes are most relevant to divergence and to quantify this dependence.…”

Exploratory identification of predictive biomarkers in randomized trials with normal endpoints

“…In order to finally determine the relevant variables based on their permutation importance, several variable selection strategies have been proposed. In general, the Boruta method 43 and the Vita algorithm 44 can be recommended as they have shown to be well balanced in terms of sensitivity and specificity. 45 In the real-world data application, we are able to clarify which of the variables brought up by the original MOB might be truly predictive.…”

Metabolic differences between symbiont subpopulations in the deep-sea tubewormRiftia pachyptila

“…Random forests are a machine learning technique, which can be used to find the variables – here proteins – that allow to predict which datasets or samples are similar (and which ones are not; Degenhardt et al , 2019). For variable importance calculation, we employed the method from (Janitza et al , 2018) as implemented in the ranger package. This method uses a heuristic approach, where a null distribution for p-value calculation is generated based on variables with importance scores of zero or negative importance scores.…”