Interrater reliability and agreement of performance ratings: A methodological comparison

Abstract

ABSTRAC~ This paper demonstrates and compares methods for estimating the interrater reliability and interrater agreement of performance ratings. These methods can be used by applied researchers to investigate the quality of ratings gathered, for example, as criteria for a validity study, or as performance measures for selection or promotional purposes. While estimates of interrater reliability are frequently used for these purposes, indices of interrater agreement appear to be rarely reported for performance ratings. A recommended index of interrater agreement, the T index (Tinsley & Weiss, 1975), is compared to four methods of estimating interrater reliability (Pearson r, coefficient alpha, mean correlation between raters, and intraclass correlation). Subordinate and superior ratings of the performance of 100 managers were used in these analyses. The results indicated that, in general, interrater agreement and reliability among subordinates were fairly high. Interrater agreement between subordinates and superiors was moderately high; however, interrater reliability between these two rating sources was very low. The results demonstrate that interrater agreement and reliability are distinct indices and that both should be reported. Reasons are discussed as to why interrater reliability should not be reported alone.

Towards integrating acceptance and resistance research: evidence from a telecare case study

“…Three researchers independently reviewed and classified the quotes. This check-coding (Miles & Huberman, 1994) enabled us to assess the interrater reliability (Fleenor et al, 1996). After a few revisions to the code list, the three researchers' initial classifications matched at just above the 70% level.…”

A review of rubric use in higher education

“…The literature most frequently recommends two approaches to inter-rater reliability: consensus and consistency. While consensus (agreement) measures if raters assign the same score, consistency provides a measure of correlation between the scores of raters (Fleenor, Fleenor, and Grossnickle 1996). Some studies use generalisability theory to compute measurement estimates.…”

Web_2.0 and five years since: How the combination of technological and organizational initiatives influences an organization’s long-term Web_2.0 performance

“…Results of the categorization were sent to interviewees for their approval and comments. Furthermore, to increase the internal validity of the data categorization, three researchers independently categorized the interviews (Fleenor et al, 1996). The categorization results of the independent researchers were approximately 80% identical with those of the original categorization and thus deemed to be fairly valid (see Table 1).…”