Performance and estimation of the true error rate of classification rules built with additional information. An application to a cancer trial

Abstract

Classification rules that incorporate additional information usually present in discrimination problems are receiving certain attention during the last years as they perform better than the usual rules in poor discrimination problems. Fernández et al (2006) proved that these rules have a lower unconditional misclassification probability than the usual Fisher's rule but they did not consider the estimation of the conditional error probability when a training sample is given (the so-called true error rate) which is a very interesting parameter in practice.In this paper we consider the problem of estimating the true error rate of these classification rules in the classical topic of discrimination among two normal populations. We prove theoretical results on the apparent error rate of the rules that expose the need of new estimators of their true error rate. Our proposal is to also consider the additional information in the definition of the true error rate estimators. We propose four such new estimators. Two of them are defined incorporating the additional information into the leave-one-out-bootstrap. The other two are the corresponding cross-validation after bootstrap versions. We compare these new estimators with the usual ones in a simulation study and in a cancer trial application, showing the very good behavior, in terms of mean square error, of the leave-one-out bootstrap estimators that incorporate the available additional information.

dawai: AnRPackage for Discriminant Analysis with Additional Information

“…We have also extended in this paper the definitions given in previous papers (Fernández et al 2006, Conde et al 2012, Conde et al 2013 from the case of equal covariances in the different populations to the case of unequal covariances among the populations and consequently defined the corresponding restricted quadratic discriminant rules. Another novelty is the definition of estimators of the accuracy of the rules for the general more than two populations case, for restricted linear and quadratic discriminant rules, thus completing the procedures presented in those three previous papers.…”

“…As we know the classes that the observations in the test set belong to, we can estimate the true error rate. The fact that the apparent error rate increases and the true error rate decreases with γ is a typical effect for these restricted rules, see Fernández et al (2006), Conde et al (2012) and Conde et al (2013).…”

“…Finally, the third function is the err.est method for 'rlda' objects and estimates the true error rate of the restricted linear classification rules defined in an object x of class 'rlda', using the methodology developed in Conde et al (2013) and in Section 2.2:…”

“…In a series of papers, the rules appearing in that initial paper have been improved, first to deal with more general types of information expressed in terms of cones of restrictions and general covariance matrices in Fernández et al (2006) and later to the case of more than two populations in Conde et al (2012). The robustness of the rules has also been studied in Salvador, Fernández, Martín, and Rueda (2008) and good estimators of the performance of the rules (which is an essential issue in discriminant analysis) have been provided in Conde et al (2013). From now on, we will refer to these rules as restricted rules as the additional information is incorporated through restrictions on the population means.…”

“…In Section 2 we describe the statistical problem and the methodology of Fernández et al (2006), Conde et al (2012) and Conde et al (2013), which we extend to the above mentioned situations. In Section 3 we introduce the dawai package and explain some details about the main user-lever functions it includes.…”