Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric

Abstract

Data imbalance is frequently encountered in biomedical applications. Resampling techniques can be used in binary classification to tackle this issue. However such solutions are not desired when the number of samples in the small class is limited. Moreover the use of inadequate performance metrics, such as accuracy, lead to poor generalization results because the classifiers tend to predict the largest size class. One of the good approaches to deal with this issue is to optimize performance metrics that are designed to handle data imbalance. Matthews Correlation Coefficient (MCC) is widely used in Bioinformatics as a performance metric. We are interested in developing a new classifier based on the MCC metric to handle imbalanced data. We derive an optimal Bayes classifier for the MCC metric using an approach based on Frechet derivative. We show that the proposed algorithm has the nice theoretical property of consistency. Using simulated data, we verify the correctness of our optimality result by searching in the space of all possible binary classifiers. The proposed classifier is evaluated on 64 datasets from a wide range data imbalance. We compare both classification performance and CPU efficiency for three classifiers: 1) the proposed algorithm (MCC-classifier), the Bayes classifier with a default threshold (MCC-base) and imbalanced SVM (SVM-imba). The experimental evaluation shows that MCC-classifier has a close performance to SVM-imba while being simpler and more efficient.

Accuracy of deep learning to differentiate the histopathological grading of meningiomas on MR images: A preliminary study

“…For a thorough evaluation of the accuracy of our test, and to account for the strong imbalance between benign and atypical/anaplastic meningiomas in our database, the Matthews Correlation Coefficient was calculated using the Multi Class Confusion Matrix function embedded in MatLab. The Matthews Correlation Coefficient takes values in the interval [–1, 1], with 1 showing a complete agreement, –1 a complete disagreement, and 0 showing that the prediction was uncorrelated with the ground truth . A schematic representation of the workflow used in this retrospective study is reported in Fig.…”

Detection and classification of microcalcification from digital mammograms with firefly algorithm, extreme learning machine and non‐linear regression models: A comparison

“…In general, MCC is considered a balanced measure that can be used for classes that have various sizes, 31 ð27Þ Tables 12-13 and 14 give the performance analysis of the FA classifier with α = 0.1, 0.15 and 0.2, respectively. In general, MCC is considered a balanced measure that can be used for classes that have various sizes, 31 ð27Þ Tables 12-13 and 14 give the performance analysis of the FA classifier with α = 0.1, 0.15 and 0.2, respectively.…”

Aging Detection of Electrical Point Machines Based on Support Vector Data Description

“…Then, we measured the performance of the classification with the Matthews correlation coefficient (MCC) [20]. MCC is a metric to evaluate the classifier performance, which has been recently acknowledged as an elective measure in the machine learning community.…”