Effect of rater training on rater accuracy: Levels-of-processing theory and social facilitation theory perspectives.

Abstract

We use levels-of-processing theory and social facilitation theory to explain the effect of training format and group size on distance and correlation accuracy, leniency-severity, halo, retention of training and pretraining information, and subject arousal. The training factor included frame-of-reference (FOR) training, information only (INFO) training, and no training (NOT). Group size was n = 1, n = 6, and n = 12, respectively. A total of 108 subjects, randomly assigned to one of nine Training × Group Size conditions, viewed and rated videotaped lectures. Results indicated that FOR training effected improved retention of training information, improved distance accuracy, and less halo over INFO training or no training (p < .05). Group size significantly affected the retention of pretraining information but not the retention of training information. Discussion centers on the components of FOR training responsible for improved rating accuracy and error, the relation between rating knowledge and rating accuracy, and implications for future research.

Application of Latent Trait Models to Identifying Substantively Interesting Raters

“…Several of the studies that compare rater behaviors have based those comparisons on the amount of rater experience or loose definitions of "expertise," and these studies have generally supported the notions that raters who have different amounts of experience or expertise behave differently when assigning ratings (Breland & Jones, 1984;Huot, 1988;Pula & Huot, 1993;Vaughan, 1991). A few studies have examined the association between rater characteristics and rating contexts and somewhat general measures of rating quality such as rating "accuracy" or "reliability" without focusing on specific patterns or trends in the ratings themselves, and the results of these studies support the notion that rating quality may differ when ratings are assigned by raters who have different experiences (Athey & McIntyre, 1987;Clauser, Clyman, & Swanson, 1999;Shohamy, Gordon, & Kraemer, 1992;Wolfe, 1997;Wolfe, Kao, & Ranney, 1997).…”

Preliminary Investigation of the Impact of a Web-Based Module on Direct Behavior Rating Accuracy

“…Using frame-of-reference training, participants review the performance of targets and then aspects of the performance central to evaluation are discussed relative to each performance dimension. Frame-ofreference training has been found to be effective in improving rating accuracy and is often the method of choice when conducting rater training (Athey & McIntyre, 1987;Roch, Woehr, Mishra, & Kieszczynska, 2012;Woehr & Huffcutt, 1994).…”

The Current State of Performance Appraisal Research and Practice: Concerns, Directions, and Implications

“…They concluded that rater training should emphasize observation and decision making processes rather than simply error reduction. In a laboratory study using student subjects and video-taped lectures, Athey and McIntyre (1987) showed that frame-of-reference training improved retention of information, improved accuracy, and decreased halo. A recent literature review of twenty-four rater training studies suggested the training methods best suited for reducing halo, reducing leniency, and improving accuracy .…”