Fuzzy kappa for the agreement measure of fuzzy classifications

2010

J Classif

Section: Introductionmentioning

confidence: 99%

A Formal Proof of a Paradox Associated with Cohen’s Kappa

2010

J Classif

“…Steinley, 2004). The popularity of κ has led to the development of many extensions, e.g., multirater kappa (Fleiss, 1971;Conger, 1980), or fuzzy kappa (Dou, Ren, Wu, Ruan, Chen, Bloyet, & Constans, 2007). However, several authors have identified difficulties or paradoxes with κ's interpretation (see, e.g., Brennan & Prediger, 1981, Feinstein & Cicchetti, 1990, or Byrt, Bishop & Carlin, 1993 and the references therein).…”

Section: Introductionmentioning

confidence: 99%

Inequalities Between Kappa and Kappa-Like Statistics for k×k Tables

2009

Psychometrika

“…Cohen's κ is by far the most popular statistic of interrater agreement for nominal categories (Landis and Koch 1977;Schouten 1980Schouten , 1982Schouten , 1986Brennan and Prediger 1981;Zwick 1988;Hsu and Field 2003;Artstein and Poesio 2005;De Mast 2007;Gwet 2008;Warrens 2010a,b,c). The popularity of κ has led to the development of many extensions (Banerjee et al 1999;Kraemer 1980;Kraemer et al 2002), including, e.g., weighted versions for ordinal categories (Cohen 1968;Vanbelle and Albert 2009;Warrens 2010d) and a version for fuzzy data (Dou et al 2007).…”

mentioning

confidence: 99%

Inequalities between multi-rater kappas

2010

Adv Data Anal Classif

187

121

The paper presents inequalities between four descriptive statistics that have been used to measure the nominal agreement between two or more raters. Each of the four statistics is a function of the pairwise information. Light's kappa and Hubert's kappa are multi-rater versions of Cohen's kappa. Fleiss' kappa is a multi-rater extension of Scott's pi, whereas Randolph's kappa generalizes Bennett et al. S to multiple raters. While a consistent ordering between the numerical values of these agreement measures has frequently been observed in practice, there is thus far no theoretical proof of a general ordering inequality among these measures. It is proved that Fleiss' kappa is a lower bound of Hubert's kappa and Randolph's kappa, and that Randolph's kappa is an upper bound of Hubert's kappa and Light's kappa if all pairwise agreement tables are weakly marginal symmetric or if all raters assign a certain minimum proportion of the objects to a specified category.