Rater Source Effects Are Alive and Well After All

Abstract

Recent research has questioned the importance of rater perspective effects on multisource performance ratings (MSPRs). Although making a valuable contribution, we hypothesize that this research has obscured evidence for systematic rater source effects as a result of misspecified models of the structure of multisource performance ratings and inappropriate analytic methods. Accordingly, this study provides a reexamination of the impact of rater source on multisource performance ratings by presenting a set of confirmatory factor analyses of two large samples of multisource performance rating data in which source effects are modeled in the form of second‐order factors. Hierarchical confirmatory factor analysis of both samples revealed that the structure of multisource performance ratings can be characterized by general performance, dimensional performance, idiosyncratic rater, and source factors, and that source factors explain (much) more variance in multisource performance ratings whereas general performance explains (much) less variance than was previously believed. These results reinforce the value of collecting performance data from raters occupying different organizational levels and have important implications for research and practice.

A Meta-Analysis of the Relationship Between Rater Liking and Performance Ratings

“…Evidence supports that raters from different levels provide unique performance information (Hoffman et al, 2010;Hoffman & Woehr, 2009), implicating overlapping but distinct nomological networks for different sources' ratings. Yet the preponderance of past likingperformance rating research has used a single source of ratings, prohibiting a direct comparison across sources.…”

“…Multiple sources of systematic variance characterize performance ratings (Murphy & DeShon, 2000). Variance related to actual ratee performance is considered construct valid, or true score variance; nonsystematic variance is conceptualized as error; and systematic variance that is unrelated to true score is conceptualized as bias (Hoffman, Lance, Bynum, & Gentry, 2010;Lance, Hoffman, Gentry, & Baranik, 2008). Although the preponderance of research has conceptualized the overlap between rater liking and performance ratings as indicative of rater bias, others have suggested that the relationship between liking and performance ratings reflects "true" differences in ratee performance (Allen & Rush, 1998;Lefkowitz, 2000;Varma et al, 1996).…”

How displaying empathic concern may differentially predict career derailment potential for women and men leaders in Australia

“…Leaders must satisfy multiple stakeholders (Tsui & Ashford, 1994;Tsui, Ashford, St. Clair, & Xin, 1995; see also the ecological perspective of multisource ratings : Hoffman, Lance, Bynum, & Gentry, 2010;Lance, Baxter, & Mahan, 2006;Lance, Hoffman, Gentry, & Baranik, 2008), and they must understand what signals their boss and their peers 3 in particular attend to when evaluating them. Relying solely on top-down (i.e., boss) ratings of career derailment potential is somewhat inconsistent with the practical realities of today's socially complex workplace (Gentry & Sosik, 2010) with different stakeholders evaluating leaders based on their own specific expectations of what leaders should do and how they should behave (Tsui & Ashford, 1994;Tsui et al, 1995).…”

Why the Qualms With Qualitative? Utilizing Qualitative Methods in 360° Feedback

“…Although much debate has revolved around the meaning in rating distinctions across the various rater categories (e.g., Hoffman, Lance, Bynum, & Gentry, 2010), the inherent value of 360°feedback is that it provides the focal participant with a behaviorally based assessment of his/her performance that is less likely to be criterion deficient (compared with a single-source method). In researching all of the early works purporting the use of 360°feedback, all researchers suggested that utilizing a single-source methodology (i.e., manager ratings) for understanding performance was likely to be criterion deficient (e.g., Edwards & Ewen, 1996;Murphy & Cleveland, 1995).…”