Geometry- and Accuracy-Preserving Random Forest Proximities

Abstract

Random forests are considered one of the best out-of-the-box classification and regression algorithms due to their high level of predictive performance with relatively little tuning. Pairwise proximities can be computed from a trained random forest which measure the similarity between data points relative to the supervised task. Random forest proximities have been used in many applications including the identification of variable importance, data imputation, outlier detection, and data visualization. However, existing definitions of random forest proximities do not accurately reflect the data geometry learned by the random forest. In this paper, we introduce a novel definition of random forest proximities called Random Forest-Geometry-and Accuracy-Preserving proximities (RF-GAP). We prove that the proximity-weighted sum (regression) or majority vote (classification) using RF-GAP exactly match the out-of-bag random forest prediction, thus capturing the data geometry learned by the random forest. We empirically show that this improved geometric representation outperforms traditional random forest proximities in tasks such as data imputation and provides outlier detection and visualization results consistent with the learned data geometry.

Decision Tree Ensembles Utilizing Multivariate Splits Are Effective at Investigating Beta Diversity in Medically Relevant 16S Amplicon Sequencing Data

“…In previous work, we have shown that ensembles built using such splits do a better job of modeling the underlying patterns in data ( 25 ). This is also well supported by other work in the field ( 15 , 17 , 24 , 41 ). By using LANDMark, TreeOrdination also minimizes the impact of noisy features through randomization (bootstrapping of training data at each node, random selection of features, and models) and regularization (most models selected for splitting are L1 or L2 regularized) ( 21 , 25 ).…”

“…Unlike statistical models, machine learning models tend not to assume anything about the underlying distribution of each feature ( 4 , 5 ). Furthermore, some machine learning models, such as random forest (RF) and related classifiers, are capable of identifying dependencies between features without the need for the user to explicitly include these dependencies in the model ( 11 , 14 – 17 ). One ability, arguably underused, inherent to this class of models is that they can be used in an “unsupervised” manner to learn a dissimilarity function ( 15 , 18 , 19 ).…”

“…One ability, arguably underused, inherent to this class of models is that they can be used in an “unsupervised” manner to learn a dissimilarity function ( 15 , 18 , 19 ). This is known as metric learning, and the learned dissimilarity function can be used to replace a more traditional method (such as the Jaccard distance or Bray-Curtis dissimilarity) when investigating beta diversity ( 17 , 20 ). This approach is also advantageous since it learns to remove the influence of uninformative features ( 21 ).…”

“…Unsupervised random forests have previously been used to discover similar cell populations in single-cell transcriptome sequencing (RNA-seq) data, identify different classes of renal cell carcinoma tumors, and study the underlying structure of a population using shared genetic variations ( 20 , 22 , 23 ). If this approach is applied to amplicon sequencing data, it may be possible to simultaneously visualize the differences between communities while also identifying which features contribute most to the placement of each community within the projected space ( 17 , 20 ). However, random forests do suffer an important limitation: the decision trees used to construct the forest make axis-orthogonal cuts.…”