On Association Coefficients, Correction for Chance, and Correction for Maximum Value

Warrens, Matthijs J.

doi:10.14355/jmmf.2013.0204.01

Cited by 5 publications

(5 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, the use of Kappa is not only extended but accepted, and its pitfalls are overcome by considering the marginal distributions and using weighted alternatives, as, for example the one suggested by Cohen ([15]), PABAK or other alternatives ([35] and [36]).…”

Section: Introductionmentioning

confidence: 99%

Why Cohen’s Kappa should be avoided as performance measure in classification

2019

View full text Add to dashboard Cite

We show that Cohen’s Kappa and Matthews Correlation Coefficient (MCC), both extended and contrasted measures of performance in multi-class classification, are correlated in most situations, albeit can differ in others. Indeed, although in the symmetric case both match, we consider different unbalanced situations in which Kappa exhibits an undesired behaviour, i.e. a worse classifier gets higher Kappa score, differing qualitatively from that of MCC. The debate about the incoherence in the behaviour of Kappa revolves around the convenience, or not, of using a relative metric, which makes the interpretation of its values difficult. We extend these concerns by showing that its pitfalls can go even further. Through experimentation, we present a novel approach to this topic. We carry on a comprehensive study that identifies an scenario in which the contradictory behaviour among MCC and Kappa emerges. Specifically, we find out that when there is a decrease to zero of the entropy of the elements out of the diagonal of the confusion matrix associated to a classifier, the discrepancy between Kappa and MCC rise, pointing to an anomalous performance of the former. We believe that this finding disables Kappa to be used in general as a performance measure to compare classifiers.

show abstract

Section: Introductionmentioning

confidence: 99%

Why Cohen’s Kappa should be avoided as performance measure in classification

2019

View full text Add to dashboard Cite

show abstract

“…In statistics and data analysis various functions have been used to normalize certain measures of similarity or association [3][4][5][6][7]. Sets of these normalization functions also form semigroups under function composition [8].…”

Section: Resultsmentioning

confidence: 99%

Semigroups of data normalization functions

Warrens¹

2016

imf

Self Cite

View full text Add to dashboard Cite

Variable centering and scaling are functions that are typically used in data normalization. Various properties of centering and scaling functions are presented. It is shown that if we use two centering functions (or scaling functions) successively, the result depends on the order in which the functions are applied: the second function always cancels the centering or scaling of the first function. Furthermore, it is shown that if we use a centering and a scaling function successively, the result does not depend on the order in which the functions are applied. Moreover, certain sets of normalization functions turn out to be semigroups.

show abstract

“…In a way, an inequality, if it exists, formalizes that two coefficients tend to measure agreement between the observers in a similar way, but to a different extent. For example, between the observed agreement and the kappa coefficients we have the inequalities P o > κ and D i > κ i for any category i (Warrens 2008(Warrens , 2010a(Warrens , 2013b. The inequalities show that, for any data, the chance-corrected coefficients will always produce a lower value than the corresponding, original (uncorrected) coefficients.…”

Section: Relationships To Other Coefficientsmentioning

confidence: 91%

“…Coefficient (3) quantifies the agreement between the observers on category i. Coefficient (3) corrects the Dice coefficient in (1) for that type of agreement that arises from chance alone (Warrens 2008(Warrens , 2010a(Warrens , 2013b. Coefficient (3) has value 1 when there is perfect agreement between the two observers on category i (then π i+ = π +i ), and 0 when agreement on category i is equal to that expected under statistical independence (i.e.…”

Section: Kappa Coefficientsmentioning

confidence: 99%

Properties of Bangdiwala’s B

Warrens

Raadt

2018

Adv Data Anal Classif

Self Cite

View full text Add to dashboard Cite

Cohen's kappa is the most widely used coefficient for assessing interobserver agreement on a nominal scale. An alternative coefficient for quantifying agreement between two observers is Bangdiwala's B. To provide a proper interpretation of an agreement coefficient one must first understand its meaning. Properties of the kappa coefficient have been extensively studied and are well documented. Properties of coefficient B have been studied, but not extensively. In this paper, various new properties of B are presented. Category B-coefficients are defined that are the basic building blocks of B. It is studied how coefficient B, Cohen's kappa, the observed agreement and associated category coefficients may be related. It turns out that the relationships between the coefficients are quite different for 2 × 2 tables than for agreement tables with three or more categories.

show abstract

On Association Coefficients, Correction for Chance, and Correction for Maximum Value

Cited by 5 publications

References 43 publications

Why Cohen’s Kappa should be avoided as performance measure in classification

Why Cohen’s Kappa should be avoided as performance measure in classification

Semigroups of data normalization functions

Properties of Bangdiwala’s B

Contact Info

Product

Resources

About