A Formal Proof of a Paradox Associated with Cohen’s Kappa

Abstract

Suppose two judges each classify a group of objects into one of several nominal categories. It has been observed in the literature that, for fixed observed agreement between the judges, Cohen's kappa penalizes judges with similar marginals compared to judges who produce different marginals. This paper presents a formal proof of this phenomenon.

Cohen’s Linearly Weighted Kappa is a Weighted Average of 2×2 Kappas

“…The kappa coefficient (Cohen, 1960;Brennan & Prediger, 1981;Zwick, 1988;Hsu & Field, 2003;Warrens 2008aWarrens , 2008bWarrens , 2010aWarrens , 2010bWarrens , 2010d, denoted by κ, is widely used as a descriptive statistic for summarizing the cross-classification of two variables with the same unordered categories. Originally proposed as a measure of agreement between two raters classifying subjects into mutually exclusive categories, Cohen's κ has been applied to square cross-classifications encountered in psychometrics, educational measurement, epidemiology (Jakobsson & Westergren, 2005), diagnostic imaging (Kundel & Polansky, 2003), map comparison (Visser & de Nijs, 2006), and content analysis (Krippendorff, 2004;Popping, 2010).…”

Cohen’s quadratically weighted kappa is higher than linearly weighted kappa for tridiagonal agreement tables

“…The kappa coefficient (denoted by κ) is a widely used descriptive statistic for summarizing two nominal variables with identical categories [2,5,19,20,21,22,25,26]. Cohen's κ was originally proposed as a measure of agreement between two raters (observers) who rate each of the same sample of objects (individuals, observations) on a nominal scale with n ∈ N ≥2 mutually exclusive categories.…”

Reply

“…We agree that prevalence affects standard Cohen's kappa calculations 1 and we are aware of the criticisms for using standard Cohen's kappa 2 . We have re-analyzed our data using Gwet et al's first order agreement coefficient (AC1) 3 .…”