Adaptive Confidence Intervals for the Test Error in Classification

Abstract

The estimated test error of a learned classifier is the most commonly reported measure of classifier performance. However, constructing a high quality point estimator of the test error has proved to be very difficult. Furthermore, common interval estimators (e.g. confidence intervals) are based on the point estimator of the test error and thus inherit all the difficulties associated with the point estimation problem. As a result, these confidence intervals do not reliably deliver nominal coverage. In contrast we directly construct the confidence interval by use of smooth datadependent upper and lower bounds on the test error. We prove that for linear classifiers, the proposed confidence interval automatically adapts to the non-smoothness of the test error, is consistent under fixed and local alternatives, and does not require that the Bayes classifier be linear. Moreover, the method provides nominal coverage on a suite of test problems using a range of classification algorithms and sample sizes.

Estimating Individualized Treatment Rules Using Outcome Weighted Learning

“…Due to high heterogeneities among individuals, there may be large variations in the estimated treatment rules across different training sets. Laber & Murphy (2011) construct an adaptive confidence interval for the test error under the non-regular framework. Confidence intervals for value functions help us determine whether essential differences exist among different decision rules.…”

Dynamic Treatment Regimes

“…Unfortunately even in an unweighted classification problem, constructing a CI for the test error is di cult due to the inherent non-smoothness; standard methods like normal approximation or usual bootstrap fail. Laber and Murphy [83] developed a method for constructing such CIs using smooth data-dependent upper and lower bounds on the test error; this method is similar to the ACI method described in Section 4.2. While intuitively one can expect that this method could be successfully adapted for the Value of an estimated DTR, more targeted research is needed to extend and fine-tune the procedure to the current setting.…”

Comment

“…Laber and Murphy (2011) use an interesting non-regular framework to examine the behavior of the test error for a wide class of linear classifiers. The non-regular framework highlights why existing confidence intervals do not work well for small-sized samples.…”

“…Using the non-regular framework, the authors show that the scaled and centered test error can be decomposed into two components, corresponding to points that are on the optimal decision boundary and points that are not (see Laber and Murphy 2011, Eq. 6).…”

“… Inspired by the non-regular framework studied in Laber and Murphy (2011), we propose a family of adaptive classifiers. We discuss briefly their asymptotic properties and show that under the non-regular framework these classifiers have an “oracle property,” and consequently have smaller asymptotic variance and smaller asymptotic test error variance than those of the original classifier.…”