Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation.

Abstract

Undergraduate subjects possessing normative or idiosyncratic rating standards were given frame-ofreference training, rater-error training, training that controlled for structural similarities between frame-of-reference training and rater-error training, or null control training. Hypothesized pretest differences that normative raters are more accurate than idiosyncratic raters were not found. However, when data were collapsed across rating aptitude, different trainings were found to improve different measures of accuracy. Frame-of-reference trainees were most accurate on stereotype accuracy and differential accuracy, rater-error trainees were most accurate on elevation, and all groups improved on differential elevation. Results are discussed in relation to the role of rater aptitude in frame-of-reference training and the future of rater-training programs.Recent performance appraisal studies on rater training have focused on frame-of-reference training (FOR) and rater-error training (RET; e.g., Hedge &Kavanaugh, 1988;Pulakos, 1984). These studies have generally concluded that FOR is superior to RET. As a result, researchers have begun to isolate aspects of the content of FOR that lead to increased accuracy in performance ratings (Athey & Mclntyre, 1987;Sulsky & Day, 1992). However, there are several limitations in the FOR research that raise questions about the general superiority of FOR. Our purpose in this study was to address these limitations and to assess whether FOR is best for all types of rating accuracy.From a historical perspective, FOR evolved from Bernardin's work (Bernardin & Buckley, 1981;Bernardin & Pence, 1980) on rater training. Bernardin and Pence noted that traditional RET stresses that certain rating distributions are more desirable than others and that RET facilitates the learning of a new response set, which may result in lower mean ratings (less leniency) and lower scale intercorrelations (less halo), but which may also lower levels of accuracy. Indeed, they found that RET led to less accurate ratings than those obtained from a group of untrained raters. Bernardin and Pence concluded that there was a need to develop new rater-training programs that increase rating accuracy, and Bernardin and Buckley (1981) proposed FOR as an alternative training strategy.As originally proposed, FOR initially involves the identification of raters who possess idiosyncratic performance standards.

Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings

“…Clips were a combination of both actual third-grade classroom footage, as well as contrived second-grade classroom footage wherein students displayed varying levels of specific behaviors per researcher instructions. The use of short video clips of behavior as stimulus material was considered to be acceptable given the history of use of such an approach in prior rater training studies (e.g., LeBel et al, 2010;McIntyre, Smith, & Hassett, 1984;Stamoulis & Hauenstein, 1993;Woehr, 1994).…”

Preliminary Investigation of the Impact of a Web-Based Module on Direct Behavior Rating Accuracy

“…Training in behavioral assessment methods has been proposed as one way to improve the accuracy of ratings and minimize the impact of rater factors (Stamoulis & Hauenstein, 1993). However, training guidelines are seldom included or reviewed in commercially available rating scales (Floyd & Bose, 2003).…”

Performance management and assessment: Methods for improved rater accuracy and employee goal setting

“…Later research found that raters who received FOR training were the most accurate in discriminating among behavioral dimensions of job performance (stereotype accuracy) and discriminating among ratees within dimensions (differential accuracy), while those who received RET were most accurate on elevation (level of ratings), and raters who received either RET or FOR training improved their ability to discriminate among ratees (Stamoulis & Hauenstein, 1993). Therefore, rater error training would be more useful when the primary goal of appraisal is to distinguish accurately among workers, while frame-of-reference training would be more useful when the primary goal of the appraisal system is to provide accurate developmental feedback to workers.…”

“…measures of accuracy. Stamoulis & Hauenstein (1993) Frame-of-reference training helps distinguish Frame-of-reference and rater error training between dimensions; rater error training can be beneficial for different purposes. increases accuracy of rating favorability; both (See Table II.)…”