The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation

Abstract

Evaluating binary classifications is a pivotal task in statistics and machine learning, because it can influence decisions in multiple areas, including for example prognosis or therapies of patients in critical conditions. The scientific community has not agreed on a general-purpose statistical indicator for evaluating two-class confusion matrices (having true positives, true negatives, false positives, and false negatives) yet, even if advantages of the Matthews correlation coefficient (MCC) over accuracy and F1 score have already been shown.In this manuscript, we reaffirm that MCC is a robust metric that summarizes the classifier performance in a single value, if positive and negative cases are of equal importance. We compare MCC to other metrics which value positive and negative cases equally: balanced accuracy (BA), bookmaker informedness (BM), and markedness (MK). We explain the mathematical relationships between MCC and these indicators, then show some use cases and a bioinformatics scenario where these metrics disagree and where MCC generates a more informative response.Additionally, we describe three exceptions where BM can be more appropriate: analyzing classifications where dataset prevalence is unrepresentative, comparing classifiers on different datasets, and assessing the random guessing level of a classifier. Except in these cases, we believe that MCC is the most informative among the single metrics discussed, and suggest it as standard measure for scientists of all fields. A Matthews correlation coefficient close to +1, in fact, means having high values for all the other confusion matrix metrics. The same cannot be said for balanced accuracy, markedness, bookmaker informedness, accuracy and F1 score.

The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation

“…Full-size  DOI: 10.7717/peerj-cs.623/fig-1 (Matthews, 1975) in binary classification (Chicco & Jurman, 2020;Chicco, Tötsch & Jurman, 2021;Tötsch & Hoffmann, 2021;Chicco, Starovoitov & Jurman, 2021;Chicco, Warrens & Jurman, 2021), R-squared generates a high score only if the regression is able to correctly classify most of the elements of each class. In this example, the regression fails to classify all the elements of the 4 class and of the 5 class, so we believe a good metric would communicate this key-message.…”

Classification of Mental Stress Using CNN-LSTM Algorithms with Electrocardiogram Signals

“…Table 2 shows the accuracy, sensitivity, specificity, precision, and negative predictive values obtained to evaluate the classification model's performance using formulas ( 9)-( 13) [26][27][28]. Formula (1) defines accuracy and indicates the probability of accurately classifying all under stress and without stress conditions.…”

Molecular insights on ABL kinase activation using tree-based machine learning models and molecular docking

“…Starting from five fingerprints, 479 AdaBoost (AB) models, 549 decision tree (DT) models, and 2.903 random forest (RF) models were built and validated for each fingerprint and endpoint, summing 59,640 machine learning models (a boxplot analysis for each model and fingerprints are displayed in the Supplementary Figure S1). The cross-validation MCC value was selected to evaluate the model’s performance due to their capability of classifying the performance by a single value, comprising all parameters of a confusion matrix [ 88 , 89 ]. In this sense, if the model had the higher CV MCC and EXT MCC, then it was selected.…”