Estimating within-group interrater reliability with and without response bias.

Abstract

This article presents methods for assessing agreement among the judgments made by a single group of judges on a single variable m regard to a single target For example, the group of judges could be editorial consultants, members of an assessment center, or members of a team. The single target could be a manuscript, a lower level manager, or a team. The variable on which the target is judged could be overall pubhshabihty in the case of the manuscript, managerial potential for the lower level manager, or team cooperativeness for the team The methods presented are based on new procedures for estimating interrater reliability For situations such as the above, these procedures are shown to furnish more accurate and mterpretable estimates of agreement than estimates provided by procedures commonly used to estimate agreement, consistency, or interrater reliability In addition, the proposed methods include processes for controlling for the spurious influences of response biases (e.g., positive leniency, social desirability) on estimates of interrater reliability.Many occasions arise in research and practice when it is useful to have an estimate of interrater reliability for judgments of a single target by one set of judges. Examples include the needs to estimate interrater reliability among judges' ratings of (a) the level of performance indicated by a potential anchor for a Behaviorally Anchored Rating Scale (BARS) in the development phases of a BARS, and (b) the overall "pubhshabihty" of a manuscript submitted for journal review. In these examples, the "variable" consists of a single item, with a rating scale such as a 7-point performance scale. It is also helpful to have an index of interrater reliability when scores on a variable consist of means taken over items that

Does safety climate moderate the influence of staffing adequacy and work conditions on nurse injuries?

“…These data were aggregated to represent measurement at the nursing unit level. Justification for data aggregation was based on values equal to or greater than .70 for the r wg statistic, which estimates within-group agreement (James, Demaree, & Wolf;1984;Lindell, Brandt, & Whitney, 1999). We also estimated the proportion of variance explained by group membership using the intraclass correlation coefficient or ICC(1) and mean rater reliability of the aggregated data using ICC(2).…”

Conhecimento Científico sobre Liderança: Uma Análise Bibliométrica do Acervo do The Leadership Quarterly

“…Finalmente, vale notar os trabalhos de Bliese (2000) e James et al (1984) próximos no mapa, e o de Podsakoff et al (2003). Eles discorrem sobre a demonstração de concordância nas avaliações de líderes por seus subordinados em estudos empíricos quantitativos com grupos como unidades de análise, e sobre os vieses em análises estatísticas quando as variáveis são levantadas com base na mesma fonte e método de mensuração nas surveys, mais uma vez ressaltando a marcante presença de abordagens metodológicas de orientação positivista como base no desenvolvimento das pesquisas publicadas no LQ.…”

A multi-level analysis of team climate and interpersonal exchange relationships at work

“…This agreement means that reliability of group climate takes into account differences within branches relative to differences between branches. Generally, an r wg greater than .70 is desirable; hence, the higher the value of r wg , the stronger within-group agreement of the construct is reflected (James, Demaree, & Wolf, 1984).…”