Rethinking Interrater Agreement

Reassessing the Elephant, Part 2

“…I have found the single‐parameter estimates of agreement to be too opaque and complicated to be of much use, and I created a way to look within a set of ratings at the scale in more detail. The new kappa was previewed in Assessment Update (Eubanks 2016) and published in an article within the context of writing assessment (Eubanks 2017). As the statistics in the latter article show, an unusual aspect to our writing ratings is that the upper end of the scale (ready to graduate) shows more agreement than the lower end (not doing college‐level work).…”

“…P art 1 of this article ( Eubanks 2019 ) described a method of crowdsourcing learning outcomes from course instructors. This half is an exposition of qualities of the data set that has resulted, with a focus on student writing.…”