A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target

Abstract

The commonly used form of r wg. (J) can display irregular behavior, so four variants of this index were examined. An alternative index, r* wg. J, is recommended. This index is an inverse linear function of the ratio of the average obtained variance to the variance of uniformly distributed random error. r* wg.Jis superficially similar to Cronbach’s α, but careful examination confirms that r* wg.Jis an index of agreement, not reliability. Based on an examination of the small-sample behavior of r wgand r* wg.J, sample sizes of 10 or more raters are recommended.

Does safety climate moderate the influence of staffing adequacy and work conditions on nurse injuries?

“…These data were aggregated to represent measurement at the nursing unit level. Justification for data aggregation was based on values equal to or greater than .70 for the r wg statistic, which estimates within-group agreement (James, Demaree, & Wolf;1984;Lindell, Brandt, & Whitney, 1999). We also estimated the proportion of variance explained by group membership using the intraclass correlation coefficient or ICC(1) and mean rater reliability of the aggregated data using ICC(2).…”

Classification of Group Potency Levels of Software Development Student Teams

“…The Group potency average for all groups was 3.63 (SD=0.7146). The agreement within-group members was calculated by the Interrater Agreement (IRA) measure [28]. The IRA average was 77.00%.…”

Changes in the Status of Native Brook Trout on Laurel Hill, Southwestern Pennsylvania

“…LEAD<-rwg.j(bhr2000[,2:12] Note that Lindell and colleagues (Lindell & Brandt, 19972000;Lindell, Brandt & Whitney, 1999) have raised concerns about the mathematical underpinnings of the r wg(j) formula. Specifically, they note that this formula is based upon the Spearman-Brown reliability estimator.…”