Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap

Kim, Jihyun

doi:10.1016/j.csda.2009.04.009

Cited by 655 publications

(382 citation statements)

References 12 publications

Supporting

Mentioning

362

Contrasting

Unclassified

Order By: Relevance

“…In order to avoid performance bias due to an arbitrary split of the labeled data, we resplit the labeled data into the training and the test sets for a total of 30 splits and averaged the models performance. This repeated cross-valdiation approach [46] has been used in similar studies to estimate the sensivity and average accuracy of the model using different training and test data [15,17,24,26,29]. We used a three-fold design where the labeled dataset was split into three groups of crowns where two groups were used to train the model and one group was reserved to test the model.…”

Section: Assessment Of Svm Classifiermentioning

confidence: 99%

Tree Species Abundance Predictions in a Tropical Agricultural Landscape with a Supervised Classification Model and Imbalanced Data

et al. 2016

View full text Add to dashboard Cite

Abstract:Mapping species through classification of imaging spectroscopy data is facilitating research to understand tree species distributions at increasingly greater spatial scales. Classification requires a dataset of field observations matched to the image, which will often reflect natural species distributions, resulting in an imbalanced dataset with many samples for common species and few samples for less common species. Despite the high prevalence of imbalanced datasets in multiclass species predictions, the effect on species prediction accuracy and landscape species abundance has not yet been quantified. First, we trained and assessed the accuracy of a support vector machine (SVM) model with a highly imbalanced dataset of 20 tropical species and one mixed-species class of 24 species identified in a hyperspectral image mosaic (350-2500 nm) of Panamanian farmland and secondary forest fragments. The model, with an overall accuracy of 62%˘2.3% and F-score of 59%˘2.7%, was applied to the full image mosaic (23,000 ha at a 2-m resolution) to produce a species prediction map, which suggested that this tropical agricultural landscape is more diverse than what has been presented in field-based studies. Second, we quantified the effect of class imbalance on model accuracy. Model assessment showed a trend where species with more samples were consistently over predicted while species with fewer samples were under predicted. Standardizing sample size reduced model accuracy, but also reduced the level of species over-and under-prediction. This study advances operational species mapping of diverse tropical landscapes by detailing the effect of imbalanced data on classification accuracy and providing estimates of tree species abundance in an agricultural landscape. Species maps using data and methods presented here can be used in landscape analyses of species distributions to understand human or environmental effects, in addition to focusing conservation efforts in areas with high tree cover and diversity.

show abstract

Section: Assessment Of Svm Classifiermentioning

confidence: 99%

Tree Species Abundance Predictions in a Tropical Agricultural Landscape with a Supervised Classification Model and Imbalanced Data

et al. 2016

View full text Add to dashboard Cite

show abstract

“…This can be done by using a repeated cross-validation algorithm, which, in comparison with other estimators such as repeated hold-out and bootstrap, was shown to be an adequate estimator of accuracy (Kohavi 1995) or related classification error rates (Kim 2009). The penalized accuracy estimations presented in the following sections correspond to the cross-validated estimates, derived using repeated tenfold cross-validation.…”

Section: Absolute Goodness-of-fit: Penalized Accuracy and Cross-validmentioning

confidence: 99%

A multivariate generalized linear tsunami fragility model for Kesennuma City based on maximum flow depths, velocities and debris impact, with evaluation of predictive accuracy

et al. 2015

View full text Add to dashboard Cite

The recent losses caused by the unprecedented 2011 Great East Japan Tsunami disaster have stimulated further research efforts, notably in the mechanisms and probabilistic determination of tsunami-induced damage, in order to provide the necessary information for future risk assessment and mitigation. The stochastic approach typically adopts fragility functions, which express the probability that a building will reach or exceed a predefined damage level usually for one, sometimes several measures of tsunami intensity. However, improvements in the derivation of fragility functions are still needed in order to yield reliable predictions of tsunami damage to buildings. In particular, extensive disaggregated databases, as well as measures of tsunami intensity beyond the commonly used tsunami flow depth should be used to potentially capture variations in the data which have not been explained by previous models. This study proposes to derive fragility functions with additional intensity measures for the city of Kesennuma, which was extensively damaged during the 2011 tsunami and for which a large and disaggregated dataset of building damage is available. In addition to the surveyed tsunami flow depth, the numerically estimated flow velocities as well as a binary indicator of debris impact are included in the model and used simultaneously to estimate building damage probabilities. Following the recently proposed methodology for fragility estimation based on generalized linear models, which overcomes the shortcomings of classic linear regression in fragility analyses, ordinal regression is applied and the reliability of the model estimates is assessed using a proposed penalized accuracy measure, more suitable than the traditional classification error rate for ordinal models. In order to assess the predictive power of the model, penalized accuracy is estimated through a repeated tenfold cross-validation scheme. For the first time, multivariate tsunami fragility functions are derived and represented in the 123Nat Hazards (2015) 79:2073-2099 DOI 10.1007/s11069-015-1947 form of fragility surfaces. The results show that the model is able to predict tsunami damage with satisfactory predictive accuracy and that debris impact is a crucial factor in the determination of building collapse probabilities.

show abstract

“…To obtain reliable estimates of the prediction error, quite extensive resampling is required in both the cross-validation and bootstrap procedures (see e.g. Kim, 2009). However, in large scale problems, recalculating a robust fit a large number of times becomes very time consuming.…”

Section: Introductionmentioning

confidence: 99%

Fast robust estimation of prediction error based on resampling

Khan

Aelst

Zamar

2010

Computational Statistics & Data Analysis

View full text Add to dashboard Cite

Robust estimators of the prediction error of a linear model are proposed. The estimators are based on the resampling techniques cross-validation and bootstrap. The robustness of the prediction error estimators is obtained by robustly estimating the regression parameters of the linear model and by trimming the largest prediction errors. To avoid the recalculation of timeconsuming robust regression estimates, fast approximations for the robust estimates of the resampled data are used. This leads to time efficient and robust estimators of prediction error.

show abstract

Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap

Cited by 655 publications

References 12 publications

Tree Species Abundance Predictions in a Tropical Agricultural Landscape with a Supervised Classification Model and Imbalanced Data

Tree Species Abundance Predictions in a Tropical Agricultural Landscape with a Supervised Classification Model and Imbalanced Data

A multivariate generalized linear tsunami fragility model for Kesennuma City based on maximum flow depths, velocities and debris impact, with evaluation of predictive accuracy

Fast robust estimation of prediction error based on resampling

Contact Info

Product

Resources

About