Sample size planning for classification models

Abstract

In biospectroscopy, suitably annotated and statistically independent samples (e.g. patients, batches, etc.) for classifier training and testing are scarce and costly. Learning curves show the model performance as function of the training sample size and can help to determine the sample size needed to train good classifiers. However, building a good model is actually not enough: the performance must also be proven. We discuss learning curves for typical small sample size situations with 5-25 independent samples per class. Although the classification models achieve acceptable performance, the learning curve can be completely masked by the random testing uncertainty due to the equally limited test sample size. In consequence, we determine test sample sizes necessary to achieve reasonable precision in the validation and find that 75-100 samples will usually be needed to test a good but not perfect classifier. Such a data set will then allow refined sample size planning on the basis of the achieved performance. We also demonstrate how to calculate necessary sample sizes in order to show the superiority of one classifier over another: this often requires hundreds of statistically independent test samples or is even theoretically impossible. We demonstrate our findings with a data set of ca. 2550 Raman spectra of single cells (five classes: erythrocytes, leukocytes and three tumour cell lines BT-20, MCF-7 and OCI-AML3) as well as by an extensive simulation that allows precise determination of the actual performance of the models in question.

Spectral pre and post processing for infrared and Raman spectroscopy of biological tissues and cells

“…Statistical significance is a critical consideration, and when patient diagnosis is the outcome, misclassification has serious consequences. Beleites et al, have carried out a study examining the effects of sample size on multivariate classifier models for clinical biospectroscopy [55]. It is demonstrated that, while learning curves for dataset sizes common to small scale academic studies can indicate acceptable performance, the model testing is itself limited by the dataset size and that datasets of 75-100 samples are required to produce "a good but not perfect classifier".…”

Design and first applications of a flexible Raman micro-spectroscopic system for biological imaging

“…The concentration of proteins differs as well. Obtained data can be used for purposes of cell identification [7,38] or planning of further biospectroscopic experiments [5].…”

“…The tube lens L 4 focuses the parallel beam of Raman signal. Depending on the mode of operation, the Raman signal is delivered to the SPG either via MMF (confocal Raman) or free-space via mirrors M 3,4 and lenses L 5,6 .…”

Active Learning as a Way of Increasing Accuracy

“…It is a general phrase in literature that more samples in training gives more successful classification result [3]. But studies also focus on cost and time effect of classification process [4], [5].…”

“…Not only the quality, but also the sample size selected in training step plays important role on the results. More selected samples in training generate more accurate classification models hence, logically; the result success should be increased as well [3]. This theory can't be generalized over all datasets but mostly data set classification models give more successful separation with more training samples.…”