Chance-corrected measures of reliability and validity in K K tables

Abstract

When studying the degree of overall agreement between the nominal responses of two raters, it is customary to use the coefficient kappa. A more detailed analysis requires the evaluation of the degree of agreement category by category, and this is carried out in two different ways: using the value of kappa in the collapsed table for each category or using the agreement index for each category (proportion of agreements observed). Both indices have disadvantages: the former is sensitive to marginal totals; the latter is not chance corrected; and neither distinguishes the case where one of the two raters is a gold standard (an expert) from the case where neither rater is a gold standard. This article suggests five chance-corrected indices which are not sensitive to marginal totals and which differ depending on whether there is a standard rater. The article also justifies the reason for poor performance of kappa when the two marginal totals are unbalanced (especially if they are so in opposite directions) and the reason for its good performance when analysing the various 2 x 2 tables obtained by the collapse of a wider table.

Validation of Self-Reported Sleep Against Actigraphy

“…If marginal totals are very small or very unbalanced, the resulting kappa can be paradoxically high or low as compared with the proportion of observed agreement. 30 , 31 Delta is an alternative chance-corrected measure of validity that is not sensitive to marginal totals but will be similar to kappa when marginal totals are not excessively unbalanced. 31 Because of the small numbers in this study, delta values were calculated in addition to kappa values using the program written by Martin and Femia.…”

“…31 Because of the small numbers in this study, delta values were calculated in addition to kappa values using the program written by Martin and Femia. 31 …”

Antiretroviral Therapy as a Factor Protective against Anal Dysplasia in HIV-Infected Males Who Have Sex with Males

“…The degree of concordance between cytology, PCR of HR-HPV, and biopsy results was analyzed using the Kappa index. The results of the test were evaluated using the classification of Landis and Koch in which a value of k<0.20 would be considered poor; 0.21 to 0.40 weak; 0.41 to 0.60 moderate; 0.61 to 0.80 good; and 0.81 to 1.00 very good [21].…”

Prospective comparison of optic versus blind endoscopic ultrasound in staging esophageal cancer

“…However, there was a higher incidence of M1a disease in patients staged with the blind endoprobe (p = 0.007), arguably reflecting larger tumour burden in patients with stenotic tumours, and consistent with the comparatively more advanced T stages. It is regrettable that previous studies do not always report weighted kappa values, as they have long been proven to accurately predict the strength of agreement between the nominal responses of two raters [29,30]. Moreover, this method has proven to be accurate when applied to medical staging investigations [31,32], especially when compared with histopathology [33].…”