Peer assessment using comparative and absolute judgement

Abstract

Citation: JONES, I. and WHEADON, C., 2015. Peer assessment using comparative and absolute judgement. Studies in Educational Evaluation, 47, pp. 93-101.Additional Information:• This paper was accepted for publication in the journal Studies in Edu- ABSTRACTPeer assessment exercises yield varied reliability and validity. To maximise reliability and validity, the literature recommends adopting various design principles including the use of explicit assessment criteria. Counter to this literature, we report a peer assessment exercise in which criteria were deliberately avoided yet acceptable reliability and validity were achieved. Based on this finding, we make two arguments.First, the comparative judgement approach adopted can be applied successfully in different contexts, including higher education and secondary school. Second, the success was due to this approach; an alternative technique based on absolute judgement yielded poor reliability and validity. We conclude that sound outcomes are achievable without assessment criteria, but success depends on how the peer assessment activity is designed.

Applying Criteria to Examples or Learning by Comparison: Effects on Students' Evaluative Judgment and Performance in Writing

“…It has been established that, even in the absence of evaluation criteria, the process of comparative judgment is easier and leads to more accurate evaluations of quality than absolute judgments in which products are evaluated one by one (Laming, 2004;Gill and Bramley, 2013). In addition, a recent meta-analysis shows that peers are as reliable in making comparative judgments as expert assessors (Verhavert et al, submitted), and that their judgments largely correspond (Jones and Alcock, 2014;Jones and Wheadon, 2015;Bouwer et al, 2018).…”

Assessment by Comparative Judgement: An Application to Secondary Statistics and English in New Zealand

“…Mathematics GPA was a significant predictor of comparative judgement scores, b = 0.052, p = 0.002, and English GPA was also a significant predictor, b = 0.044, p = 0.001. This was not expected because previous research conducted in the UK has consistently found that mathematics, but not English, attainment is a significant predictor mathematics comparative judgement scores (Jones et al 2013;Jones and Wheadon 2015;Jones and Karadeniz 2016). To understand the relationship between the comparative judgement scores and GPAs further we investigated the correlations.…”

Untitled

“…(1) rank-order and parameter value statistics for all the items being compared, (2) Rasch-model misfit statistics for items and judges that can be used to identify any potentially significant areas of disagreement, and, if it is collected, (3) comments or justifications surrounding each decision made by the judges. The final rank order has been used in a variety of ways including, but not limited to the following: assigning grades, informing teacher pedagogy and student practice, as a formative tool for improvement, and as a fractional portion of total points received for a given assignment (Bartholomew, Strimel, & Yoshikawa, 2019;Bartholomew & Yoshikawa, 2018;Jones & Wheadon, 2015;McMahon & Jones, 2015).…”