What Do Peer Evaluations Represent? A Study of Rater Consensus and Target Personality

Abstract

When professors assign group work, they assume that peer ratings are a valid source of information, but few studies have evaluated rater consensus in such ratings. We analyzed peer ratings from project teams in a second-year university course to examine consensus. Our first goal was to examine whether members of a team generally agreed on the competence of each team member. Our second goal was to test if a target’s personality traits predicted how well they were rated. Our third goal was to evaluate whether the self-rating of each student correlated with their peer rating. Data were analyzed from 130 students distributed across 21 teams (mean team size = 6.2). The sample was diverse in gender and ethnicity. Social relations model analyses showed that on average 32% of variance in peer-ratings was due to “consensus,” meaning some targets consistently received higher skill ratings than other targets did. Another 20% of the variance was due to “assimilation,” meaning some raters consistently gave higher ratings than other raters did. Thus, peer ratings reflected consensus (target effects), but also assimilation (rater effects) and noise. Among the six HEXACO traits that we examined, only conscientiousness predicted higher peer ratings, suggesting it may be beneficial to assign one highly conscientious person to every team. Lastly, there was an average correlation of.35 between target effects and self-ratings, indicating moderate self-other agreement, which suggests that students were only weakly biased in their self-ratings.

WIP: Identity-Based Bias in Undergraduate Peer Assessment

“…Other individual differences have been found to impact peer ratings such as personality [11] and 'perceiver variance' [12], in which 32% of the variance in peer ratings was due to consensus on the ratee, 20% was due to rater tendency to score high or low on certain characteristics, and other variation was residual. The residual variance could be due to relationship-specific variance, such as raters having "distinctive biases" towards ratees [12]. The residual variance could present itself as gender or racial biases, which don't account for the skill of an individual but rather the perception a rater has of an individual from perceived identities.…”