Unbiased bootstrap error estimation for linear discriminant analysis

Abstract

Convex bootstrap error estimation is a popular tool for classifier error estimation in gene expression studies. A basic question is how to determine the weight for the convex combination between the basic bootstrap estimator and the resubstitution estimator such that the resulting estimator is unbiased at finite sample sizes. The well-known 0.632 bootstrap error estimator uses asymptotic arguments to propose a fixed 0.632 weight, whereas the more recent 0.632+ bootstrap error estimator attempts to set the weight adaptively. In this paper, we study the finite sample problem in the case of linear discriminant analysis under Gaussian populations. We derive exact expressions for the weight that guarantee unbiasedness of the convex bootstrap error estimator in the univariate and multivariate cases, without making asymptotic simplifications. Using exact computation in the univariate case and an accurate approximation in the multivariate case, we obtain the required weight and show that it can deviate significantly from the constant 0.632 weight, depending on the sample size and Bayes error for the problem. The methodology is illustrated by application on data from a well-known cancer classification study.Electronic supplementary materialThe online version of this article (doi:10.1186/s13637-014-0015-0) contains supplementary material, which is available to authorized users.

ROC632: An Overview

“…These results were expected, since these curves only represent the training and test sets, respectively. The 0.632 and 0.632+ bootstrap curves had an overall similar performance, with marginally lower values for latter, since the 0.632+ estimator's performance depends on the amount of overfitting, whereas the former has a constant weight [1,14,15]. Hence, the 0.632+ estimator provided the best results with the least variance (similar results within the same condition -fixed or variable lambda) and bias (no overor underestimation) and it should thus be used in future analysis.…”

“…Finding a method for validating predictive models and obtaining an unbiased performance has been a target of discussion by multiple authors [11,16]. Although there are several approaches to estimate the error rate of a prediction rule, such as the jackknife (leave-one-out) method and cross-validation, the bootstrap method has been considered the most efficient throughout the years, as it is capable of directly assessing the variability, returns higher accuracy and it is able to calculate the variance of a point estimate of prediction error [1,14,15].…”

“…The bootstrap method separates the available data into two sets: the training set, which is used to obtain a predictive model and the test set, used to evaluate its performance [4,5,8,16]. It draws random instances with replacement (resampling) from the original dataset, which means that some sets can be used multiple times and some might not be used at all, although, typically, eventually all the data will be selected at least once [10,14,15].…”

“…However, since this method can still be biased, another estimator, the 0.632+ estimator, was created to improve it [1,14,15].…”

“…This function returns a list similar to the one described in subsection 3.1. The assessment of the boot.ROCt function included the process listed for the boot.ROC function, while varying the maximum prognostic time (pro.time) for which each variable is evaluated (from 1 to 16 years) and changing the lasso penalty (lambda1 ) from variable to fixed (15). The proportion of nearest neighbors (prop) was kept constant at a value of 0.02 during the course of the study and only 2 bootstrap iterations (N.boot) were considered.…”