Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns.

Abstract

We examined methodological and theoretical issues related to accuracy measures used as criteria in performance-rating research. First, we argued that existing operational definitions of accuracy are not all based on a common accuracy definition; we report data that show generally weak relations among different accuracy operational definitions. Second, different methods of true score development are also examined, and both methodological and theoretical limitations are explored. Given the difficulty of obtaining true scores, criteria are discussed for examining the suitability of expert ratings as surrogate true score measures. Last, the usefulness of using accuracy measures in performance-rating research is examined to highlight situations in which accuracy measures might be desirable criterion measures in rating research.

Managing “challenging” teachers

“…Appraisal methods are susceptible to many kinds of errors, and human resource experts invest much time to develop and improve the validity o f rating instruments (Barrett & Reman, 1987;Sulsky & Balzer, 1988). The use o f appraisal for academic purpose only, as in this study, does limit the range of limitations in comparison to procedures of performance improvement and personnel decision-making procedures.…”

The Current State of Performance Appraisal Research and Practice: Concerns, Directions, and Implications

“…Specifically, Schoorman (1988) reported that supervisors who had a say in the hiring decision and who viewed the applicant as favorable, subsequently tended to give lenient performance ratings while those who participated in hiring but viewed the applicant as unfavorable, tended to give more severe ratings. Finally, Sulsky and Balzer (1988) argued that accuracy in performance measurement was lacking due to poor definitions of accuracy, methodological and theoretical limitations of true score development, and the absence of a cohesive theory of performance.…”

Utility estimates of job performance as related to the Data, People, and Things parameters of work

“…For example, Bernardin and Pence (1980) reported that raters who were trained to reduce halo error actually made less accurate ratings after training. Similarly, Sulsky and Balzer (1988) discussed the uncertain relationship between inter-rater reliability, rating errors, and accuracy in performance appraisal because of our inability to know what`truth' is. Low interrater reliability is thus evidence of disagreement in making judgments.…”