Precision-Recall versus Accuracy and the Role of Large Data Sets

Abstract

Practitioners of data mining and machine learning have long observed that the imbalance of classes in a data set negatively impacts the quality of classifiers trained on that data. Numerous techniques for coping with such imbalances have been proposed, but nearly all lack any theoretical grounding. By contrast, the standard theoretical analysis of machine learning admits no dependence on the imbalance of classes at all. The basic theorems of statistical learning establish the number of examples needed to estimate the accuracy of a classifier as a function of its complexity (VC-dimension) and the confidence desired; the class imbalance does not enter these formulas anywhere. In this work, we consider the measures of classifier performance in terms of precision and recall, a measure that is widely suggested as more appropriate to the classification of imbalanced data. We observe that whenever the precision is moderately large, the worse of the precision and recall is within a small constant factor of the accuracy weighted by the class imbalance. A corollary of this observation is that a larger number of examples is necessary and sufficient to address class imbalance, a finding we also illustrate empirically.

Comparison of Transfer Learning and Conventional Machine Learning Applied to Structural Brain MRI for the Early Diagnosis and Prognosis of Alzheimer's Disease

“…Actually, there are advanced deep networks proposed recently for 3D MRI-based AD diagnosis and MCI-to-AD prediction of conversion, by using anatomical landmarks or dementia attention discovery schemes to locate those information in MRI brain regions, thus alleviating the small-sample-size problem (66)(67)(68)(69). In our work we trained from scratch a new 3D CNN achieving lower performance with respect to the other considered 2D transfer learning methods (84% vs. [89-91]% for AD vs. CN, 72% vs. [80-84]% for MCIc vs. CN, 61% vs. [69][70][71]% for MCIc vs. MCInc). However, we would like to underline that training a 3D CNN from scratch requires a huge amount of training data.…”

“…Although the estimation of the number of samples necessary to train a deep-learning classifier from scratch with good performance is still an open research problem, some studies tried to investigate this issue. According to Juba et al (70) the amount of data needed for learning depends on the complexity of the model. A rule of thumb descending from their paper is that we need 10 cases per predictor to train a simple model like a regressor, while we need 1,000 images per class to train from scratch a deep-learning classifier like a CNN.…”

DeepSurvNet: deep survival convolutional network for brain cancer survival rate classification based on histopathological images

“…Several metrics like confusion matrix [49]; the combination of precision, recall and F-score [50]; and the area under the ROC curve (AUC) [51] were used for performance evaluation of our classifiers.…”

Balancing Trade-Offs in the Detection of Primary Schools at Risk

“…However, with high class imbalance (i.e., positives negatives), the ROC curve is more informative of the identification of the majority class (schools not at risk) than of the identification of the minority class (schools at risk). Although Juba and Le (2019) show that larger data sets are necessary and sufficient for dealing with class imbalance, the number of Dutch primary schools is effectively fixed and there is limited to zero capacity for additional inspection visits.…”