Agreement of Teachers on Evaluating Assessments of Learning Progressions in English Language Arts and Mathematics

Abstract

In this study, we explored the extent to which teachers agree on the ordering and separation of levels of two different learning progressions (LPs) in English language arts (ELA) and mathematics. In a panel meeting akin to a standard‐setting procedure, we asked teachers to link the items and responses of summative educational assessments to LP levels. We evaluated four types of agreement among teachers, with test developers, and with trained raters. Although the results were quite mixed, agreement on the LP levels of mathematics items was generally better than that for ELA tasks. The agreement on the LP levels of ELA sample responses was generally better than that for mathematics. Implications of the results are discussed.

Learning Progression–Based Assessments: A Systematic Review of Student and Teacher Uses

“…Forty of the 59 sources shared data exploring how teachers or students made judgments about student learning using LP-based assessments or their corresponding score reports. Evidence in this category was primarily from teacher self-reported data and researcher observations, although two sources examined consistency of judgments (Goral & Bailey, 2019; van Rijn et al, 2018). Most reported teachers’ interpretations of their own students’ work; however, in six studies, only researcher-provided data were examined (i.e., Alonzo & Elby, 2019; Ebby & Sirinides, 2015; Jin et al, 2015; Kannan et al, 2021; Supovitz et al, 2018; van Rijn et al, 2018).…”

“…Evidence in this category was primarily from teacher self-reported data and researcher observations, although two sources examined consistency of judgments (Goral & Bailey, 2019; van Rijn et al, 2018). Most reported teachers’ interpretations of their own students’ work; however, in six studies, only researcher-provided data were examined (i.e., Alonzo & Elby, 2019; Ebby & Sirinides, 2015; Jin et al, 2015; Kannan et al, 2021; Supovitz et al, 2018; van Rijn et al, 2018). Across sources, authors noted that LP-based assessment provided a structure and language useful for interpreting student work and communicating results (e.g., Edgington, 2014; Furtak & Heredia, 2014; Stille et al, 2016; Wickstrom & Langrall, 2020; Wilson, 2014).…”

“…Teachers were reported sharing and discussing judgments in meetings (e.g., Briggs et al, 2014). Only two sources analyzed the consistency of judgments (i.e., Goral & Bailey, 2019; van Rijn et al, 2018); both suggested acceptable judgment reliability is possible, however, given studies’ small scale and diverse subject areas and assessment approaches, findings cannot be generalized. Van Rijn et al (2018) found generally good agreement between teachers and trained raters, with alignment better in English than math.…”

“…The review found limited data showing teachers using the same LP-based assessment for each purpose, even though some did provide tools to assist with both formative and summative applications (e.g., Anderson et al, 2018). Only one small-scale study examined the reliability of teacher judgments (RQ 1; van Rijn et al, 2018), highlighting the need for more work to explore how consistent judgments can be supported.…”