Assessing agreement between raters from the point of coefficients and loglinear models

Hall, Sydney; McMahon, Brian

doi:10.6339/jds.201701_15(1).0001

Cited by 8 publications

(8 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even there are many suggested weighting schemes, linear and quadratic weights are the well-known ones. For different weighting schemes in the literature, see [2].…”

Section: Inter-rater Agreement Coefficientsmentioning

confidence: 99%

See 1 more Smart Citation

How to use adjusted degree of distinguishability and inter-rater reliability simultaneously?

Yılmaz

2021

Cumhuriyet Science Journal

Self Cite

View full text Add to dashboard Cite

When the categories of a square contingency table are ordinal, weighted kappa or Gwet's AC2 coefficients are used to summarize the degree of reliability between two raters. In addition, investigating the reliability among raters, the term category distinguishability should be considered. The study aims to assess the inter-rater reliability and category distinguishability in ordinal rating scales together. The weighted kappa, AC2, and adjusted degree of distinguishability coefficients are applied to pathology data. The results are discussed over the pathologist pairs.

show abstract

“…Even there are many suggested weighting schemes, linear and quadratic weights are the well-known ones. For different weighting schemes in the literature, see [2].…”

Section: Inter-rater Agreement Coefficientsmentioning

confidence: 99%

“…Square contingency tables are occurred with the same row and column classification [1] and are frequently used in many fields, such as medicine, sociology, and behavioral sciences [2]. When working on these kinds of tables, the inter-rater reliability of row and column variables is investigated.…”

Section: Introductionmentioning

confidence: 99%

How to use adjusted degree of distinguishability and inter-rater reliability simultaneously?

Yılmaz

2021

Cumhuriyet Science Journal

Self Cite

View full text Add to dashboard Cite

show abstract

“…A number of theoretical and methodological approaches have been proposed over the years in different disciplines for the assessment of rater repeatability and/or reproducibility; these approaches can be grouped in two main families: index‐based approach and model‐based approach. The former quantifies the rater agreement level in a single number and does not provide insight into the structure and nature of agreement differences; 5‐8 the latter overcomes this criticism and models the ratings provided by each rater to each subject focusing on the association structure between repeated evaluations 9,10 …”

Section: Introductionmentioning

confidence: 99%

Robustness of κ‐type coefficients for clinical agreement

Vanacore

Pellegrino

2022

Statistics in Medicine

View full text Add to dashboard Cite

The degree of inter-rater agreement is usually assessed through 𝜅-type coefficients and the extent of agreement is then characterized by comparing the value of the adopted coefficient against a benchmark scale. Through two motivating examples, it is displayed the different behavior of some 𝜅-type coefficients due to asymmetric distribution of marginal frequencies over categories. In order to investigate the robustness of four 𝜅-type coefficients for nominal and ordinal classifications and of an inferential benchmarking procedure that, differently from straightforward benchmarking, does not neglect the influence of the experimental conditions, an extensive Monte Carlo simulation study has been conducted. The robustness has been investigated for several scenarios, differing for sample size, rating scale dimension, number of raters, frequency distribution of rater classifications, pattern of agreement across raters. Simulation results reveal an higher paradoxical behavior of Fleiss kappa and Conger kappa with ordinal rather than nominal classifications; the coefficients robustness improves with increasing sample size and number of raters for both nominal and ordinal classifications whereas robustness improves with rating scale dimension only for nominal classifications. By identifying the scenarios (ie, minimum sample size, number of raters, rating scale dimension) with acceptable robustness, this study provides guidelines about the design of robust agreement studies. K E Y W O R D Sinter-rater agreement, 𝜅-type coefficients, Monte Carlo simulation, paradoxical behavior, robustness

show abstract

“…The selection of the weighting scheme is clearly articulated by Gwet 4 (see Figure 3.6.1 in Gwet 4 ). The review of agreement measures for calculating inter-rater agreement among two or more raters are represented by Banerjee et al 5 , Gwet 4 , Yilmaz and Saracbasi 6 and recently Tran et al 3 . The number of raters can be two or more.…”

Section: Introductionmentioning

confidence: 99%

“…Our approach not only improves the accuracy and the robustness of the agreement measures against the grey zones, but also readily produces interval estimates for the degree of inter-rater agreement. The interpretation of the value of an agreement measure in terms of the strength of the agreement has not been strictly standardized yet 6 . Having a Bayesian interval estimate for the Kappa measure enhances the interpretation of the degree of agreement between the raters by providing probability limits.…”

Section: Introductionmentioning

confidence: 99%

Bayesian approaches to the weighted kappa-like inter-rater agreement measures

Tran

Demirhan

Dolgun

2021

Stat Methods Med Res

View full text Add to dashboard Cite

Inter-rater agreement measures are used to estimate the degree of agreement between two or more assessors. When the agreement table is ordinal, different weight functions that incorporate row and column scores are used along with the agreement measures. The selection of row and column scores is effectual on the estimated degree of agreement. The weighted measures are prone to the anomalies frequently seen in agreement tables such as unbalanced table structures or grey zones due to the assessment behaviour of the raters. In this study, Bayesian approaches for the estimation of inter-rater agreement measures are proposed. The Bayesian approaches make it possible to include prior information on the assessment behaviour of the raters in the analysis and impose order restrictions on the row and column scores. In this way, we improve the accuracy of the agreement measures and mitigate the impact of the anomalies in the estimation of the strength of agreement between the raters. The elicitation of prior distributions is described theoretically and practically for the Bayesian estimation of five agreement measures with three different weights using an agreement table having two grey zones. A Monte Carlo simulation study is conducted to assess the classification accuracy of the Bayesian and classical approaches for the considered agreement measures for a given level of agreement. Recommendations for the selection of the highest performing agreement measure and weight combination are made in the breakdown of the table structure and sample size.

show abstract

Assessing agreement between raters from the point of coefficients and loglinear models

Cited by 8 publications

References 50 publications

How to use adjusted degree of distinguishability and inter-rater reliability simultaneously?

How to use adjusted degree of distinguishability and inter-rater reliability simultaneously?

Robustness of κ‐type coefficients for clinical agreement

Bayesian approaches to the weighted kappa-like inter-rater agreement measures

Contact Info

Product

Resources

About