Cross-validation for selecting a model selection procedure

Yong-li, Zhang; Yang, Yuhong

doi:10.1016/j.jeconom.2015.02.006

Cited by 310 publications

(190 citation statements)

References 47 publications

Supporting

Mentioning

184

Contrasting

Unclassified

Order By: Relevance

“…19 Though common, this number will not optimize performance in all settings. 20 In general, we recommend increasing the number of folds (V ) as sample size n decreases.…”

Section: Discussionmentioning

confidence: 99%

Stacked Generalization: An Introduction to Super Learning

Naimi

Balzer

2017

Preprint

View full text Add to dashboard Cite

Stacked generalization is an ensemble method that allows researchers to combine several different prediction algorithms into one. Since its introduction in the early 1990s, the method has evolved several times into what is now known as “Super Learner”. Super Learner uses V -fold cross-validation to build the optimal weighted combination of predictions from a library of candidate algorithms. Optimality is defined by a user-specified objective function, such as minimizing mean squared error or maximizing the area under the receiver operating characteristic curve. Although relatively simple in nature, use of the Super Learner by epidemiologists has been hampered by limitations in understanding conceptual and technical details. We work step-by-step through two examples to illustrate concepts and address common concerns.

show abstract

“…19 Though common, this number will not optimize performance in all settings. 20 In general, we recommend increasing the number of folds (V ) as sample size n decreases.…”

Section: Discussionmentioning

confidence: 99%

Stacked Generalization: An Introduction to Super Learning

Naimi

Balzer

2017

Preprint

View full text Add to dashboard Cite

show abstract

“…Faced with the possibility to create a large number of predictive models, researchers depend on model selection mechanisms (Zhang & Yang, 2015). It is widely agreed upon that "significant insample evidence of predictability does not guarantee significant out-of-sample predictability" (Inoue & Kilian, 2005).…”

Section: Discussionmentioning

confidence: 99%

“…Researchers routinely create more than one predictive model while hunting for a particularly well performing one. A substantial body of research is therefore concerned with the inevitable model selection problem while developing prediction models (Zhang & Yang, 2015). To select a model, researchers can calculate and compare the predicted outcome with the actual outcome using the data that was used to fit or train the model (terminology varies between econometrics and machine learning).…”

Section: Introductionmentioning

confidence: 99%

Shedding Light on the Role of Sample Sizes and Splitting Proportions in Out-of-Sample Tests: A Monte Carlo Cross-Validation Approach

Janze

2017

Atas Da 17ª Conferência Da Associação Portuguesa De Sistemas De Informação

View full text Add to dashboard Cite

We examine whether the popular 2/3 rule-of-thumb splitting criterion used in out-of-sample evaluation of predictive econometric and machine learning models makes sense. We conduct simulations regarding the predictive performance of the logistic regression and decision tree algorithm when considering varying splitting points as well as sample sizes. Our nonexhaustive repeated random sub-sampling simulation approach known as Monte Carlo crossvalidation indicates that while the 2/3 rule-of-thumb works, there is a spectrum of different splitting proportions that yield equally compelling results. Furthermore, our results indicate that the size of the complete sample has little impact on the applicability of the 2/3 rule-ofthumb. However, our analysis reveals that when considering relatively small and relatively large training samples in relation to the sample size, the variation of the predictive accuracy can lead to misleading results. Our results are especially important for IS researchers considering the usage of out-of-sample methods for evaluating their predictive models.

show abstract

“…Each original dataset was divided into test and validation sets in proportion 9:1 with 10-fold cross-validation, which is widely accepted in data mining and machine learning community and serves as a standard procedure of validation [15]- [17].…”

Section: Methodsmentioning

confidence: 99%

Preprocessing compensation techniques for improved classification of imbalanced medical datasets

Wosiak

Karbowiak

2017

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

Abstract-The paper describes the study on the problem of applying classification techniques in medical datasets with a class imbalance. The aim of the research is to identify factors that negatively affect classification results and propose actions that may be taken to improve the performance. To alleviate the impact of uneven and complex class distribution, methods of balancing the datasets are proposed and compared. The experiments were conducted on five datasets -three binary and two multiclass. They comprise several data preprocessing methods applied on data and the classification with different techniques. The study shows that for some datasets there exists a combination of a certain preprocessing method and a classification technique which outperforms other approaches. For datasets with complex distribution or too many features the ratio of correctly predicted labels may be low regardless what resampling method and classification technique has been applied.

show abstract

Cross-validation for selecting a model selection procedure

Cited by 310 publications

References 47 publications

Stacked Generalization: An Introduction to Super Learning

Stacked Generalization: An Introduction to Super Learning

Shedding Light on the Role of Sample Sizes and Splitting Proportions in Out-of-Sample Tests: A Monte Carlo Cross-Validation Approach

Preprocessing compensation techniques for improved classification of imbalanced medical datasets

Contact Info

Product

Resources

About