Probability estimation with machine learning methods for dichotomous and multicategory outcome: Applications

Abstract

Machine learning methods are applied to three different large datasets, all dealing with probability estimation problems for dichotomous or multicategory data. Specifically, we investigate k-nearest neighbors, bagged nearest neighbors, random forests for probability estimation trees, and support vector machines with the kernels of Bessel, linear, Laplacian, and radial basis type. Comparisons are made with logistic regression. The dataset from the German Stroke Study Collaboration with dichotomous and three-category outcome variables allows, in particular, for temporal and external validation. The other two datasets are freely available from the UCI learning repository and provide dichotomous outcome variables. One of them, the Cleveland Clinic Foundation Heart Disease dataset, uses data from one clinic for training and from three clinics for external validation, while the other, the thyroid disease dataset, allows for temporal validation by separating data into training and test data by date of recruitment into study. For dichotomous outcome variables, we use receiver operating characteristics, areas under the curve values with bootstrapped 95% confidence intervals, and Hosmer-Lemeshow-type figures as comparison criteria. For dichotomous and multicategory outcomes, we calculated bootstrap Brier scores with 95% confidence intervals and also compared them through bootstrapping. In a supplement, we provide R code for performing the analyses and for random forest analyses in Random Jungle, version 2.1.0. The learning machines show promising performance over all constructed models. They are simple to apply and serve as an alternative approach to logistic or multinomial logistic regression analysis.

ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R

“…First, the R packages randomForest (Liaw and Wiener 2002), randomForestSRC (Ishwaran and Kogalur 2015) and Rborist (Seligman 2015), the C++ application Random Jungle (Schwarz et al 2010;Kruppa et al 2014b), and the R version of the new implementation ranger were run with small simulated datasets, a varying number of features p, sample size n, number of features tried for splitting (mtry) and a varying number of trees grown in the RF. In each case, the other three parameters were kept fixed to 500 trees, 1,000 samples, 1,000 features and mtry = √ p. The datasets mimic genetic data, consisting of p single nucleotide polymorphisms (SNPs) measured on n subjects.…”

“…This package is studied in greater detail in Section 5. Finally, an RF implementation optimized for analyzing high dimensional data is Random Jungle (Schwarz et al 2010;Kruppa et al 2014b). This package is only available as C++ application with library dependencies, and it is not portable to R or another statistical programming language.…”

“…Our primary aim was to develop a platform independent and modular framework for the analysis of high dimensional data with RF. Second, the package should be simple to use and available in R with a computational performance not worse than Random Jungle 2.1 (Kruppa et al 2014b). Here, we describe the new implementation, explain its usage and provide examples.…”

An Ensemble of Optimal Trees for Class Membership Probability Estimation

“…It is usually required by surgeons, oncologists, pathologists, professionals involved in internal medicine and human genetics and pediatricians (Malley et al (2012)). For instance, carrier probabilities are calculated in genetic counseling and treatment response probability is estimated in personalized medicine of every patient , Kruppa et al (2014b)). …”

Ensemble of Subset of k-Nearest Neighbours Models for Class Membership Probability Estimation

“…For example, in safety-critical domains such as surgery, oncology, internal medicine, pathology, paediatrics and human genetics, these probabilities are needed. In all the aforementioned areas, probability estimates are more useful than simple classification as they provide a measure of reliability of the decision taken about an individual (Lee et al (2010), Malley et al (2012), Kruppa et al (2012), Kruppa et al (2014aKruppa et al ( , 2014b). Machine learning techniques used mainly for classification can be used as non-parametric methods for class membership probability estimation in order to avoid the assumptions imposed in parametric models used for the estimation of these probabilities , Malley et al (2012)).…”