Examining the Validity of Adaptive Comparative Judgment for Peer Evaluation in a Design Thinking Course

Abstract

Adaptive comparative judgment (ACJ) is a holistic judgment approach used to evaluate the quality of something (e.g., student work) in which individuals are presented with pairs of work and select the better item from each pair. This approach has demonstrated high levels of reliability with less bias than other approaches, hence providing accurate values in summative and formative assessment in educational settings. Though ACJ itself has demonstrated significantly high reliability levels, relatively few studies have investigated the validity of peer-evaluated ACJ in the context of design thinking. This study explored peer-evaluation, facilitated through ACJ, in terms of construct validity and criterion validity (concurrent validity and predictive validity) in the context of a design thinking course. Using ACJ, undergraduate students (n = 597) who took a design thinking course during Spring 2019 were invited to evaluate design point-of-view (POV) statements written by their peers. As a result of this ACJ exercise, each POV statement attained a specific parameter value, which reflects the quality of POV statements. In order to examine the construct validity, researchers conducted a content analysis, comparing the contents of the 10 POV statements with highest scores (parameter values) and the 10 POV statements with the lowest scores (parameter values)—as derived from the ACJ session. For the criterion validity, we studied the relationship between peer-evaluated ACJ and grader’s rubric-based grading. To study the concurrent validity, we investigated the correlation between peer-evaluated ACJ parameter values and grades assigned by course instructors for the same POV writing task. Then, predictive validity was studied by exploring if peer-evaluated ACJ of POV statements were predictive of students’ grades on the final project. Results showed that the contents of the statements with the highest parameter values were of better quality compared to the statements with the lowest parameter values. Therefore, peer-evaluated ACJ showed construct validity. Also, though peer-evaluated ACJ did not show concurrent validity, it did show moderate predictive validity.

Comparative Judgement

“…In comparison to the issue of reliability, the validity of Comparative Judgement has much less attention, although there has recently been a proliferation of research focused on this issue (e.g., Chambers & Cunningham, 2022;Humphry & Heldsinger 2019a;Lesterhuis et al, 2022;Mentzer, Lee, & Bartholomew, 2021;van Daal, Lesterhuis, Coertjens, van de Kamp, Donche, & De Maeyer, 2017;van Daal et al, 2019). Typically, the method has been found to have moderate to high levels of concurrent validity with other more established methods like rubric-based ratings (e.g., Attali et al, 2014;Heldsinger & Humphry, 2010, 2013McGrane et al, 2018).…”