How to use adjusted degree of distinguishability and inter-rater reliability simultaneously?

Abstract

When the categories of a square contingency table are ordinal, weighted kappa or Gwet's AC2 coefficients are used to summarize the degree of reliability between two raters. In addition, investigating the reliability among raters, the term category distinguishability should be considered. The study aims to assess the inter-rater reliability and category distinguishability in ordinal rating scales together. The weighted kappa, AC2, and adjusted degree of distinguishability coefficients are applied to pathology data. The results are discussed over the pathologist pairs.