We examined methodological and theoretical issues related to accuracy measures used as criteria in performance-rating research. First, we argued that existing operational definitions of accuracy are not all based on a common accuracy definition; we report data that show generally weak relations among different accuracy operational definitions. Second, different methods of true score development are also examined, and both methodological and theoretical limitations are explored. Given the difficulty of obtaining true scores, criteria are discussed for examining the suitability of expert ratings as surrogate true score measures. Last, the usefulness of using accuracy measures in performance-rating research is examined to highlight situations in which accuracy measures might be desirable criterion measures in rating research.

The different conceptual and operational definitions of halo are reviewed, and problems when using halo as a dependent measure in performance rating research and practice are pointed out. Four major points are emphasized: (a) There is no agreed on conceptual definition of halo; (b) the different conceptual definitions of halo are not systematically related to different operational definitions (i.e., measures) of halo; (c) halo measures may be poor indexes of rating quality in that different halo measures are not strongly interrelated and halo measures are not related to measures of rating validity or accuracy; and (d) although halo may be a poor measure of rating quality, it may or may not be an important measure of the rating process. The utility of assessing halo to determine the psychometric quality of rating data is questioned. Halo may be more appropriately used as a measure to study cognitive processing, rather than as a measure of performance rating outcome.Since Wells (1907) and Thorndike (1920), psychologists have (a) provided definitions and interpretations of halo (e.g., Cooper, 1981), (b) expanded and revised operational measures of halo (e.g., Pulakos, Schmitt, & Ostroff, 1986), (c) examined the relation between halo and other measures of rating quality (e.g., Murphy & Balzer, 1989), and (d) predicted and evaluated the effects of halo on performance ratings (e.g., Kozlowski & Kirsch, 1987). Hundreds of papers have been published or presented addressing these issues. Despite the massive work in this area, no recent reviews have integrated or critically examined this growing literature.We examine several conceptual and operational problems when using halo as a dependent variable in performance rating research. Our review and critique are organized around four major points: (a) There is no agreed on conceptual definition of halo; (b) different conceptual definitions of halo are not systematically related to different operational definitions (i.e., measures); (c) halo measures are not strongly interrelated or related to measures of rating validity or accuracy; and (d) although halo may be a poor measure of rating quality, it may or may not be an important measure of aspects of the rating process.

The purpose of this study was to demonstrate how examining the bivariate correlations between items in self-report measures can assist in differentiating between possible common method variance vs. model specification errors. Specifically, social desirability was viewed as either a possible source of common method variance or as a theoretically meaningful construct that should be included in the model of interest (i.e., a specification error). In the first instance, LISREL was used, and the level of correlation between measures of social desirability and measures of the five constructs of interest was manipulated. These results provided some insight as to when one needs to be concerned about the possible "common variance effects" on the structural model. In the second instance, the correlations between measures of social desirability and the measures of only two constructs of interest were again manipulated. These analyses illustrated the point at which the omission of social desirability as a theoretically relevant variable began to result in a poor fit of the structural model.

We considered the effects of frame-of-reference (FOR) training on raters' ability to correctly classify ratee performance as well as their ability to recognize previously observed behaviors. The purpose was to examine the cognitive changes associated with FOR training to better understand why such training generally improves rating accuracy. We trained college students (N = 94) using either FOR or control procedures, had them observe three managers on videotape, and rate the managers on three performance dimensions. Results supported our hypotheses that, compared with control training, FOR training led to better rating accuracy and better classification accuracy. Also consistent with predictions, FOR training resulted in lower decision criteria (i.e., higher bias) and lower behavioral accuracy on a recognition memory task involving impression-consistent behaviors. The implications of these results are discussed, particularly in terms of the ability of FOR-trained raters to provide accurate performance feedback to ratees.

The authors investigated the effects of frame-of-reference (FOR) training on various indexes of distance and correlational accuracy under alternative time delays. One hundred fifty subjects were assigned randomly to either FOR-or control-(i.e., minimal) training conditions, with 1 of 3 time delays: (a) no delay between training, observation, and rating; (b) ratings performed 2 days following training and ratee observations; or (c) ratee observations and ratings completed 2 days following training. Hypotheses were proposed predicting specific relationships between accuracy, recall memory, and learning, depending on the delay period. Overall, results supported the categorization perspective on FOR-training effectiveness; however, different results were obtained depending on the type of accuracy index and time delay. The authors discuss the implications of these findings in terms of how they relate to the conceptual distinction between distance and correlational accuracy and to the role of on-line, memory-based, and inference-memory-based processing in the ratings of FORtrained raters.Previous research has established that frame-of-reference (FOR) training results in performance ratings that are significantly more accurate in comparison with other types of (or no) training (e.g.,