Overall indices for assessing agreement among multiple raters

Abstract

The need to assess agreement exists in various clinical studies where quantifying inter-rater reliability is of great importance. Use of unscaled agreement indices, such as total deviation index and coverage probability (CP), is recommended for two main reasons: (i) they are intuitive in a sense that interpretations are tied to the original measurement unit; (ii) practitioners can readily determine whether the agreement is satisfactory by directly comparing the value of the index to a prespecified tolerable CP or absolute difference. However, the unscaled indices were only defined in the context of comparing two raters or multiple raters that assume homogeneity of variances across raters. In this paper, we introduce a set of overall indices based on the root mean square of pairwise differences that are unscaled and can be used to evaluate agreement among multiple raters that often exhibit heterogeneous measurement processes in practice. Furthermore, we propose another overall agreement index based on the root mean square of pairwise differences that is scaled and extends the concept of the recently proposed relative area under CP curve in the presence of multiple raters. We present the definitions of overall indices and propose inference procedures in which bootstrap methods are used for the estimation of standard errors. We assess the performance of the proposed approach and demonstrate its superiority over the existing methods when raters exhibit heterogeneous measurement processes using simulation studies. Finally, we demonstrate the application of our methods using a renal study.

Evaluate the reliability of the apprenticeship in the first year of medical school: towards a reliable first level ultrasound examination

“…This boundary called total deviation index ( TDI ) is proportional to the square root of the MSD and it is a performance measurement. A satisfactory agreement may require a large CP or equivalently a small TDI [18,20,21].…”

Using multiple agreement methods for continuous repeated measures data: a tutorial for practitioners

“…Additionally, Barnhart [12] and Barnhart et al [11] describe an interesting method involving the use of generalised estimating equations to provide a non-parametric estimate of the CP. Recently Jang et al [35] have proposed a new set of agreement indices suitable for contexts in which there are multiple raters and heterogeneous variances.…”

Assessment model of university innovation ecosystems in ASEAN

“…For each of the universities selected, two representatives were recruited to complete the innovation rubric for the university. Two representatives were necessary to address respondent bias through the use of a multi-rater approach (Jeong et al, 2018). These representatives were selected using expert sampling (Chaudhuri, 2019), meaning that they were purposely selected based on their knowledge of and involvement in the university's innovation process.…”

“…The solution arrived at was the use of a standard measure for each level, along with the use of multiple raters for each of the institutions. This strategy was chosen because the use of multiple raters is an acknowledged way to offset bias in assessment practices (Jeong et al, 2018).…”

“…For both service levels, the process will begin with multiple university experts completing the AIUEA assessment based on the assessment rubric. It is recommended that at least three experts be included for the purposes of bias reduction through multiple raters (Jeong et al, 2018), but the number of experts can vary depending on the university and its situation. Following assessment completion, scores are tabulated and recommendations are generated based on identified problem areas.…”