Relevance of rater-ratee acquaintance in the validity and reliability of ratings.

Abstract

Unacquainted Se worked in three-man groups under relevant (mathematical tasks) and irrelevant (socializing) acquaintance conditions. The Ss rated one another OB scales that defined several cognitive skills, They were also rated on these same scales by observers who were dependent on visual information, exclusively, and were unacquainted with the group members or the specific nature of the tasks being performed. As hypothesized, group members under the relevant acquaintance condition achieved consistently good validity for all three cognitive areas--with the best validity for ratings of math anility. Validity under the irrelevant acquaintance condition was nil on all scales. Observers, surprisingly, achieved significant validity (although at lower levels than participating group members) for ratings under the relevant acquaintance condition. Levels of inter-and intrarater reliability were not associated with levels oi validity under the various rating conditions. Efforts devoted to improvement of the rating process have focused primarily upon content, construction, and format of the rating instrument. Only limited attention has been paid to the nature of the rater-ratee relationship-s-In terms of the conditions under which raters make their observations or the availability of behavioral cues relevant to the characteristics being judged.The importance of this aspect of rating has not gone unnoticed in general discussions by such writers as Thorndike and Hagen (1961) who conclude that "the ideal rater is the person who has had a great deal of opportunity to observe the person being rated in situations in which he would be likely to show the qualities on which ratings are desired [Ch. 13]." Burtt (1942) has similarly stressed "the conditions under which the ratee has been observed" as affecting the accuracy of rating and suggests that, in addition to requiring information regarding how long or how well the rater knows the ratee, one might also request information regarding the circumstances under which he was known. In his mathernatico-deductive Theory of Rating, Wherry ( 1952) derives a theorem which specifies that "raters will vary in the accuracy 1 The research reported in this paper is based, in part, on data utilized for a doctoral dissertation submitted to the Ohio State University.

Exploring the Relationship Between Interrater Correlations and Validity of Peer Ratings

“…Above this level, validity was stable and almost did not change as interrater correlation increased. This finding, together with other studies (Borman, 1975;Buckner, 1959;Freeberg, 1969;Weekley & Gier, 1989), cast doubt on the assertion that interrater correlation in the field of performance rating is a proper measurement of reliability.According to classical measurement theory, validity and reliability are related. Cronbach (1970) stated, "The error portion of the test score will not correlate with any criterion.…”

Self–other rating agreement in leadership: A review

“…Another study found that ratings made by acquainted raters of briefings on research projects were more accurate than those made by unacquainted raters (Van Scotter, Moustafa, Burnett, & Michael, 2007). Freeberg (1969) found that performance ratings were more valid when raters possessed task-relevant acquaintance with the ratee than when they possessed merely task-irrelevant acquaintance (e.g., socializing). Kingstrom and Mainstone (1985) found that ratees with whom supervisors had established relatively high task acquaintance (familiarity with the employee's job behavior) and personal acquaintance received more favorable performance ratings than other ratees.…”

Validating the Gre Analytical Ability Measure Against Faculty Ratings of Analytical Reasoning Skills

“…The first six scales were designed to assess the kinds of reasoning skills that emerged from the factor analysis of faculty ratings of the importance of various reasoning skills and are described in Table 2. In addition, a seventh scale was included to obtain an indication of each rater's familiarity with each student, since familiarity is likely to affect the quality of ratings and therefore the relationship of ratings to test performance (Freeberg, 1969;Landy & Farr, 1980). Moreover, the importance of obtaining some indication of raters' acquaintance with ratees has been reinforced recently in the new Standards for Educational and Psychological Testing (AERA, APA, NCME, 1985), which state that when criteria are composed of rater judgments, the degree of knowledge that raters have concerning ratee performance should be reported.…”

“…Previous research has suggested that the nature of raters' contact with ratees may affect the quality of ratings, hence their relationship to other variables (e.g., Freeberg, 1969;Landy & Farr, 1980). In providing their ratings in this study, faculty were also asked to describe how much opportunity (significantly less, slightly less, neither more nor less, slightly more, or significantly more compared with other students) they had to observe/judge the extent to which the students they rated possessed the kinds of analytical skills of interest.…”