The Effect of Similarity on Performance Ratings and Interrater Agreement

Abstract

Although training and use of multiple raters have been suggested to minimize demographic and individual biases on performance evaluations (Latham & Wexley, 1981), use of multiple raters often results in low interrater reliability which compounds the difficulty of establishing performance rating validity. The present study investigated the performance ratings made by the peers and supervisors of correction's officers, i.e., prison guards. Attention wasfocused on two issues: (1) the effects of rater and ratee personal, attitudinal, and relationship variables on performance ratings, and (2) examining performance "perspective differences" as an explanation of rating agreement. Results indicated that supervisor-subordinate educational similarity was significantly related to subordinate's performance rating. Supervisor-subordinate agreement on job dimension importance and subordinate age similarity with peer raters were also marginally related to performance ratings. The findings are discussed from the perspectives of shared cognitive schemas ofjob behaviors and practical implications.

Similarity attraction in learning contexts: An empirical study

“…Additionally, the similarity hypothesis has been tested beyond laboratory settings where the controlled conditions and the use of bogus strangers limited the generalizability of initial findings. These investigations indicate that similarity accounts for decisions in such organizational settings as performance evaluations (Strauss, Barrick, & Connerley, 2001;Zalesny & Kirsch, 1989), selections (Phillips & Phillips, 1996), negotiations (Graham, Kim, Lin, & Robinson, 1988), and leader-follower relationships (Ashkanasy & O'Connor, 1997). Together, these investigations provide support for the pervasive effect of similarity attraction.…”

“…Hence, introducing subjects' scores separately into analyses is conceptually inconsistent with the essence of our endogenous variable. As such, mimicking previous studies in similarity (e.g., Zalesny & Kirsch, 1989), we relied on subjects' absolutedifferences scores to operationalize similarity agreeableness. We reversed values of differences so that scores approaching zero indicated T-T' s dissimilarity.…”

Value Congruence and Charismatic Leadership in CEO–Top Manager Relationships: An Empirical Investigation

“…|D| is a profile similarity index calculated by taking the sum of the absolute differences between profile elements (e.g., between CEO and top management team member ratings of each value) -thus lower levels of |D| represent greater congruence between the values of CEOs and each subordinate. |D| has been used to represent congruence by many researchers of organizational phenomena (Bernardin and Alvares, 1975;Greene and Organ, 1973;Johnson and Graen, 1973;Zalesny and Kirsch, 1989) including value congruence in leader-subordinate relationships (Ashkanasy and O'Connor, 1997;Engle and Lord, 1997). Meglino and Ravlin (1998) have advised that a profile similarity index that measures distance between value profiles is essential in the measurement of value congruence.…”

Problems With the Use of Profile Similarity Indices in the Study of Congruence in Organizational Research

“…D, or the square root of D2 (e.g., Drazin & Van de Ven, 1985;Govindarajan, 1989;Gresov, 1989;Sparrow, 1989;Vancouver & Schmitt, 1991; Vandenberg & Scarpello, 1990; Wexley & Pulakos, 1983; White, Crino, & Hatfield, 1985) is also nondirectional but, unlike D2, yields a geometric interpretation, representing the Euclidean distance between the two entities in k-dimensional space, where k represents the number of elements involved. ID1 represents the sum of absolute differences between profile elements (e.g., Dougherty & Pritchard, 1985; Meglino, Ravlin, & Adkins, 1992;Zalesny & Kirsch, 1989). IDJ is nondirectional but, unlike D2, assigns equal weight to differences of increasing magnitude (e.g., a difference of three units on one element is equivalent to a difference of one unit on three elements).…”