Simulating and estimating agreement in the presence of multiple raters and covariates

Abstract

Cohen's and Fleiss's kappa are popular estimators for assessing agreement among two and multiple raters, respectively, for a binary response. While additional methods have been developed to account for multiple raters and covariates, they are not always applicable, rarely used, and none simplify to Cohen's kappa. Furthermore, there are no methods to simulate Bernoulli observations under the kappa agreement structure such that the developed methods could be adequately assessed. This manuscript overcomes these shortfalls. First, we developed a model-based estimator for kappa that accommodates multiple raters and covariates through a generalized linear mixed model and encompasses Cohen's kappa as a special case. Second, we created a framework to simulate dependent Bernoulli observations that upholds all 2-tuple pair of rater's kappa agreement structure and includes covariates. We used this framework to assess our method when kappa was nonzero. Simulations showed that Cohen's and Fleiss's kappa estimates were inflated unlike our model-based kappa. We analyzed an Alzheimer's disease neuroimaging study and the classic cervical cancer pathology study. The proposed model-based kappa and advancement in simulation methodology demonstrates that the popular approaches of Cohen's and Fleiss's kappa are poised to yield invalid conclusions while our work overcomes shortfalls, leading to improved inferences.

Analysis of Variation of Tin Filter on Noise Values in CT Scan Mastoid Protocol Using CT Scan Single Source: Phantom Study

“…Overall, the subjective assessment of the two radiologists agreed that the mastoid CT scan protocol with Sn100 could provide better and more informative image quality results so that it could assist radiologists in making a diagnosis. The resulting kappa value is 0.75 with a significant value of 0.013, indicating that the coefficient value indicates a correlation or level of agreement at a moderate level (McKenzie & Mahnken, 2023). subjectively, namely noise, sharpness, anatomical boundaries, and the smallest anatomical parts, namely the cochlear apical turn, malleus incus, and cochlear nerve opening.…”