A comparison of tight generalization error bounds

Kääriäinen, Matti; Langford, John

doi:10.1145/1102351.1102403

Cited by 13 publications

(13 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For an excellent introduction see the tutorial by Langford (2005). There are few studies reported that compare the performance of different confidence intervals but recently the empirical applicability of five different upper bounds was compared on 29 real world data sets (Kaariainen and Langford, 2005). One of the most successful bounds was the binomial test bound recently proposed by Langford which is based on the binomial distribution.…”

Section: Alternatives To CV and Btsmentioning

confidence: 99%

Cross-validation and bootstrapping are unreliable in small sample classification

Isaksson

Wallman

Göransson

et al. 2008

Pattern Recognition Letters

164

View full text Add to dashboard Cite

a b s t r a c tThe interest in statistical classification for critical applications such as diagnoses of patient samples based on supervised learning is rapidly growing. To gain acceptance in applications where the subsequent decisions have serious consequences, e.g. choice of cancer therapy, any such decision support system must come with a reliable performance estimate. Tailored for small sample problems, cross-validation (CV) and bootstrapping (BTS) have been the most commonly used methods to determine such estimates in virtually all branches of science for the last 20 years. Here, we address the often overlooked fact that the uncertainty in a point estimate obtained with CV and BTS is unknown and quite large for small sample classification problems encountered in biomedical applications and elsewhere. To avoid this fundamental problem of employing CV and BTS, until improved alternatives have been established, we suggest that the final classification performance always should be reported in the form of a Bayesian confidence interval obtained from a simple holdout test or using some other method that yields conservative measures of the uncertainty.

show abstract

Section: Alternatives To CV and Btsmentioning

confidence: 99%

Cross-validation and bootstrapping are unreliable in small sample classification

Isaksson

Wallman

Göransson

et al. 2008

Pattern Recognition Letters

164

View full text Add to dashboard Cite

show abstract

“…However, the bounds using such analogy are either very loose are hard to compute in practice. Prediction error bounds on the other hand are easier to compute for practical application of the prediction theory (Kääriäinen and Langford, 2005). Hence we propose those as an extension to the routinely reported metrics such as average accuracies and receiver operating characteristic (ROC) curves.…”

Section: Methodsmentioning

confidence: 99%

“…Now, using standard properties of the binomial distribution one can obtain (Kääriäinen and Langford, 2005),…”

Section: Methodsmentioning

confidence: 99%

Penalized Likelihood Phenotyping: Unifying Voxelwise Analyses and Multi-Voxel Pattern Analyses in Neuroimaging

et al. 2013

View full text Add to dashboard Cite

Neuroimage phenotyping for psychiatric and neurological disorders is performed using voxelwise analyses also known as voxel based analyses or morphometry (VBM). A typical voxelwise analysis treats measurements at each voxel (e.g. fractional anisotropy, gray matter probability) as outcome measures to study the effects of possible explanatory variables (e.g. age, group) in a linear regression setting. Furthermore, each voxel is treated independently until the stage of correction for multiple comparisons. Recently, multi-voxel pattern analyses (MVPA), such as classification, have arisen as an alternative to VBM. The main advantage of MVPA over VBM is that the former employ multivariate methods which can account for interactions among voxels in identifying significant patterns. They also provide ways for computer-aided diagnosis and prognosis at individual subject level. However, compared to VBM, the results of MVPA are often more difficult to interpret and prone to arbitrary conclusions. In this paper, first we use penalized likelihood modeling to provide a unified framework for understanding both VBM and MVPA. We then utilize statistical learning theory to provide practical methods for interpreting the results of MVPA beyond commonly used performance metrics, such as leave-one-out-cross validation accuracy and area under the receiver operating characteristic (ROC) curve. Additionally, we demonstrate that there are challenges in MVPA when trying to obtain image phenotyping information in the form of statistical parametric maps (SPMs), which are commonly obtained from VBM, and provide a bootstrap strategy as a potential solution for generating SPMs using MVPA. This technique also allows us to maximize the use of available training data. We illustrate the empirical performance of the proposed framework using two different neuroimaging studies that pose different levels of challenge for classification using MVPA.

show abstract

“…Recent work has demonstrated Bayesian alternatives [11][12] . The ATR practitioner should always be aware of these issues and estimate and evaluate the efficacy of their mappings by these techniques.…”

Section: Intrinsic Vs Perceived Separationmentioning

confidence: 99%

Redefining automatic target recognition (ATR) performance standards

et al. 2011

View full text Add to dashboard Cite

Present descriptors for Automatic Target Recognition (ATR) performance are inadequate for use in comparing algorithms that are purported to be a solution to the problem. The use of receiver operator characteristic curves (ROCs) is a defacto standard, but they do not communicate several key performance measures, including (i) intrinsic separation between classes in the input space, (ii) the efficacy of the mapping induced by the algorithm, (iii) the complexity of the algorithmic mapping, and (iv) a measure of the generalization of the proposed solution. Previous work by Sims et. al. 2,5 has addressed the distortion of the evaluation sets to indicate an algorithm's capability (or lack thereof) for generalization and handling of unspecified cases. This paper addresses the rethinking of the summary statistics used for understanding the performance of a solution. We propose new approaches for solution characterization, allowing algorithm performance comparison in an equitable and insightful manner. This paper proffers some examples and suggests directions for new work from the community in this field.

show abstract

A comparison of tight generalization error bounds

Cited by 13 publications

References 9 publications

Cross-validation and bootstrapping are unreliable in small sample classification

Cross-validation and bootstrapping are unreliable in small sample classification

Penalized Likelihood Phenotyping: Unifying Voxelwise Analyses and Multi-Voxel Pattern Analyses in Neuroimaging

Redefining automatic target recognition (ATR) performance standards

Contact Info

Product

Resources

About