Probability estimation for large-margin classifiers

Abstract

SummaryLarge margin classifiers have proven to be effective in delivering high predictive accuracy, particularly those focusing on the decision boundaries and bypassing the requirement of estimating the class probability given input for discrimination. As a result, these classifiers may not directly yield an estimated class probability, which is of interest itself. To overcome this difficulty, this article proposes a novel method to estimate the class probability through sequential classifications, by utilising features of interval estimation of large margin classifiers.The method uses sequential classifications to bracket the class probability to yield an estimate up to the desired level of accuracy. The method is implemented for support vector machines and ψ-learning, in addition to an estimated Kullback-Leibler loss for tuning. A solution path of the method is derived for support vector machines to further reduce its computational cost. Theoretical and numerical analyses indicate that the method is highly competitive against alternatives, especially when the dimension of input greatly exceeds the sample size.Finally, an application to leukaemia data is described.

New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes

“…There is a direct analogy between hard- and soft-margin classifiers wherein hard-margin classifiers directly estimate the decision boundary and soft-margin classifiers back-out the decision boundary through conditional class probabilities. When the class probabilities are complex, hard-margin classifiers may lead to improved performance (Wang et al, 2008); likewise, when the conditional expectations are complex, directly targeting the decision rule is likely to yield improved performance. Since the proposed methods do not model the relationship between outcomes and DTRs, they may be more robust to model misspecification than statistical modeling alternatives such as Q -learning (Zhang et al, 2012b,a).…”

Probability-enhanced effective dimension reduction for classifying sparse functional data

“…Thus motivated, Lin et al (2004) proposed weighted SVM (WSVM) by weighting observations from different classes with different weights in the training process and established its Fisher consistency. Based on the WSVM's Fisher consistency, Wang et al (2008) proposed a probability estimation scheme to estimate the conditional probability for each new observation belonging to each class. Particularly inspired from this probability estimation scheme using the WSVM to address the aforementioned homogeneity issue in partitioning a binary response, Shin et al (2014) proposed a probability-enhanced dimension reduction method for multivariate data.…”

Flexible Large Margin Classifiers

“…By using different weights for observations in different classes, Wang, Shen, and Liu (2008) proposed to solve the weighted SVM where 0 < 7r < 1 and f(-) can be either linear or nonlinear with an appropriately chosen penalty J(1). For the weighted SVM (6.2), its classification boundary is shown to consistently estimate the boundary {x : p(x) = 7r}.…”

“…But the information is limited in the sense that it is only capable of telling whether the conditional probability is larger than 7r. Wang, Shen, and Liu (2008) proposed to solve weighted SVMs for different weights 7r E (0,1). Then an interval estimate for the conditional class probability is obtained.…”