Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives.

Abstract

Although peer reviewing of writing is a way to create more writing opportunities in college and university settings, the validity and reliability of peer-generated grades are a major concern. This study investigated the validity and reliability of peer-generated writing grades of 708 students across 16 different courses from 4 universities in a particular scaffolded reviewing context: Students were given guidance on peer assessment, used carefully constructed rubrics, and were provided clear incentives to take the assessment task seriously. Distinguishing between instructor and student perspectives of reliability and validity, the analyses suggest that the aggregate ratings of at least 4 peers on a piece of writing are both highly reliable and as valid as instructor ratings while (paradoxically) producing very low estimates of reliability and validity from the student perspective. The results suggest that instructor concerns about peer evaluation reliability and validity should not be a barrier to implementing peer evaluations, at least with appropriate scaffolds. Future research needs to investigate how to address student concerns about reliability and validity and to identify scaffolds that may ensure high levels of reliability and validity.

Unravelling peer assessment: Methodological, functional, and conceptual developments

“…More recently, Cho, Schunn, and Wilson (2006) found that the aggregate of at least four peer assessments were as reliable and valid as teacher assessments, whereas the reliability and validity of single peer assessments were much lower -presumably because students evaluate a subset of all teacher assessments and as a consequence develop different evaluative perspectives that are reflected in rating variability. Irrespective of the findings on the reliability of peer ratings versus teacher ratings (Cho et al, 2006;Falchikov & Goldfinch, 2000;Magin, 2001;Stefani, 1994;Topping, 2003;Zhang, Johnston, & Kilic, 2008), similarity in peer and teacher ratings provides no information as to whether the ratings affect students' subsequent performance. It is implicitly assumed that the high degree of similarity between peer and teacher ratings reflects rating fairness and that student responses to peer ratings will be similar to their responses to teacher ratings.…”

Bringing the ‘Dismal Science’ to Life: Teaching Economics Through Multimedia

“…SWoRD, a web-based peer review system, can help. It randomly assigns students within groups and creates the correct incentives for serious peer review of film projects (Cho et al, 2006;Cho and Schunn, 2007).…”

Suppressing Competition in a Computer-Supported Collaborative Learning System

“…Thus, writers return high grades to peer reviewers who give high scores or give low scores to those who give low ratings. Although peer reviewing of writing has found to be as effective as or even superior than instructor assessment in terms of writing improvement [9,19], this tit-for-tat strategy may undermine the utility of peer review.…”

“…Recently Cho, Schunn, and Wilson [9] found an interesting aspect on how students perceive peer reviews differently from instructors when the students receive good quality of peer reviews. They found that even if peer reviews are reliable and valid from the instructor's perspective, they could be perceived as unreliable and invalid from the students' perspective.…”

“…In particular, when multiple reviews are used, students may practice multiple writing with timely feedback [6]. Also it was found that peer reviews are very reliable and as valid as instructor reviews [9].…”