Handling Sparsity with Random Forests When Predicting Adverse Drug Events from Electronic Health Records

Abstract

When using electronic health record (EHR) data to build models for predicting adverse drug effects (ADEs), one is typically facing the problem of data sparsity, i.e., drugs and diagnosis codes that could be used for predicting a certain ADE are absent for most observations. For such tasks, the ability to effectively handle sparsity by the employed machine learning technique is crucial. The state-of-the-art random forest algorithm is frequently employed to handle this type of data. It has however recently been demonstrated that the algorithm is biased towards the majority class, which may result in a low predictive performance on EHR data with large numbers of sparse features. In this study, approaches to handle this problem are empirically evaluated using 14 ADE datasets and three performance metrics; F1-score, AUC and Brier score. Two resampling based techniques are investigated and compared to two baseline approaches. The experimental results indicate that, for larger forests, the resampling methods outperform the baseline approaches when considering F1-score, which is consistent with the metric being affected by class bias. The approaches perform on a similar level with respect to AUC, which can be explained by the metric not being sensitive to class bias. Finally, when considering the squared error (Brier score) of individual predictions, one of the baseline approaches turns out to be ahead of the others. A bias-variance analysis shows that this is an effect of the individual trees being more correct on average for the baseline approach and that this outweighs the expected loss from a lower variance. The main conclusion is that the suggested choice of approach to handle sparsity is highly dependent on the performance metric, or the task, of interest. If the task is to accurately assign an ADE to a patient record, a sampling based approach is recommended. If the task is to rank patients according to risk of a certain ADE, the choice of approach is of minor importance. Finally, if the task is to accurately assign probabilities for a certain ADE, then one of the baseline approaches is recommended.

Sniffing fungi – phenotyping of volatile chemical diversity inTrichodermaspecies

“…The number of variables that randomly sampled as candidates at each split (mtry) and the number of trees to grow (ntree) were tuned (grid search) using the caret package (Kuhn, 2008) to obtain the optimal predictive ability, stability and accuracy. RF analysis was performed separately on the PTR‐ToF‐MS and GC‐MS datasets, to avoid the possible influence of scale difference and sparsity difference (Karlsson & Boström, 2014).…”

Predicting United States Policy Outcomes with Random Forests

“…Logistic regression (used by Gilens and Page) includes an intrinsic, strong assumption of linearity (Pampel, 2000) (for details see Appendix). Random Forests (RFs) in contrast are flexible, non-linear classifiers (Breiman, 2001a) that can handle large numbers of sparsely-represented features such as the preferences of the 43 individual IGs in the Gilens dataset (Karlsson, 2014). In addition, RFs have natural metrics to assess the relative importance of each feature.…”

“…The final prediction of a test case is an average of the trees' predictions of that case. RFs are adept at handling sparse datasets with a large number of features (Karlsson, 2014), though removing uninformative features can improve model accuracy. The accuracies of two other flexible classifiers, XGBoost (Chen and Guestrin, 2016) (a flavor of RF) and Neural Nets, were similar to standard RFs (results not reported).…”

“…In order to identify the most important IGs, we used RF's standard Gini impurity method (Breiman and Friedman, 1984), indicated for this dataset because the features were sparse, non-categorical, and similarly scaled (in range [-2, 2]) (Karlsson, 2014). Because IGs were ranked for each Policy Domain separately, the PDs were not features.…”

“…Note that because this feature set combines one-hot and ordinal value encodings, the caveats in (Karlsson, 2014) and (Strobl et al, 2007) about distortion of feature rankings may apply. As noted previously, the RF scores for the individual PAs have the balanced accuracy target encoded into their RF scores, which effectively avoids the status quo bias intrinsic to the different policy areas.…”