Meta-analytical estimates of interrater reliability for direct supervisor performance ratings: Optimism under optimal measurement designs.

Abstract

Performance appraisal (PA) is used for various organizational purposes and is vital to human resources practices. Despite this, current estimates of PA reliability are low, leading to decades of criticism regarding the use of PA in organizational contexts. In this article, we argue that current meta-analytical interrater reliability (IRR) coefficients are underestimates and do not reflect the reliability of interest to most practitioners and researchers-the reliability of an employee's direct supervisor. To establish the reliability of direct supervisor ratings, those making PA ratings must directly supervise employee job performance instead of nonparallel rater designs (e.g., direct supervisor ratings correlated with ratings from a more senior leader). The current meta-analysis identified 22 independent samples that met this more restrictive study inclusion criterion, finding an average observed IRR of .65. We also report reliability estimates for several important contextual moderators, including whether ratings were completed in operational settings (.60) or for research purposes (.67). In sum, we argue that this study's meta-analytical IRR estimates are the best available estimates of direct supervisor reliability and should be used to guide future research and practice.

An updated meta-analysis of the interrater reliability of supervisory performance ratings.

“…Furthermore, none of the other moderators significantly influenced the interrater reliability. Although Speer et al (2023) found a higher reliability estimate for ratings made by same-level raters (i.e., two direct supervisors) than the overall reliability reported by previous studies, which seems to suggest a moderating effect of rater job levels on interrater reliability, by directly comparing same-level versus different-level raters, we found a nonsignificant difference between the two regardless of the estimator being used. Moreover, as we argued above, the Morris estimator is more suitable for meta-analyzing this literature, and it produced an overall reliability value of .65 that is comparable to Speer et al's estimate.…”

“…is the best available indicator given the limited information reported in the studies examined. While this article was under review, a meta-analysis was published (Speer et al, 2023) that looked at exactly this issue and obtained a reliability estimate of .65 for ratings made by two direct supervisors. In the discussion section, we compare our findings with those of Speer et al…”