Examining the assessment literacy required for interpreting score reports: A focus on educators of K–12 English learners

Abstract

This study investigated the assessment literacy required for K–12 educators to interpret score reports from a K–12 English language proficiency assessment. The assessment in concern is ACCESS for ELLs, which is an annual summative assessment that is delivered to nearly 2 million English learners (ELs) across 39 US states and territories. This study was conducted in two phases. In Phase 1, an online teacher survey, consisting of 15 items, was completed by 1437 participants; data were analyzed using descriptive statistics. In Phase 2, 18 educators were interviewed to obtain in-depth understanding of educators’ interpretation of the score reports. Interview data were qualitatively analyzed in terms of (1) the essential assessment literacy required for interpreting the score reports, (2) resources referred to by educators for score report interpretation, and (3) educators’ suggestions to enhance the score report and its interpretation. The findings of the study reveal the relevance of K–12 EL educators’ assessment literacy for score report interpretation. For example, educators referred to proficiency level scores the most, but experienced difficulty in understanding technical terms, such as “scale scores” or “confidence band.” The results provide implications for enhancing the quality of score reports and the development of resources to support educators’ assessment literacy.

Young learners’ voices: Towards a learner-centered approach to understanding language assessment literacy

“…Accordingly, empirical investigations of LAL have rarely incorporated learners. Studies on LAL have primarily focused on pre- and in-service language teachers (Fulcher, 2012; Kim et al, 2019; Koh et al, 2018; Lam, 2019; Lee, 2019; Levi & Inbar-Lourie, 2020; Vogt & Tsagari, 2014), higher education administrators (Baker, 2016; Deygers & Malone, 2019), policy makers (Pill & Harding, 2013), and language assessment course instructors (Jeong, 2013). Rare exceptions include Kremmel and Harding (2020) and Watanabe (2011).…”

Examining the Effects of a Real-Time, Knowledge-Aware Tool for Academic Writing Assessment

“…Moreover, another disadvantage is that the information presented in the graphs may increase the rater’s cognitive load (Sung et al, 2016), which could weaken the rater’s motivation to consistently use it (Chen & Tsao, 2021). Thus, the findings suggest that future design should improve algorithmic performance to reduce redundant information and optimize the representations of knowledge or the hierarchy of feedback to make them more accessible and digestible (Kim et al, 2020).…”

“…Interpretability of automated assessment has three major advantages. First, interpretability can improve the CSWA system’s validity since it requires explanations of the assessment criteria and the process about how the scores are generated (Kim et al, 2020; Ploegh et al, 2009). Second, interpretability helps instructors comprehend how the assessment works and thus reduces negative factors generated by assessors’ subjectivity (e.g., writing expertise, assessment experience) (Ade-Ibijola et al, 2012; Weideman, 2019).…”

Language Assessment Literacy of Teachers

“…Prior studies have found that most language teachers had insufficient LAL. Some reported that teachers incorrectly understood language assessment ( Kiomrs et al, 2011 ; Berry et al, 2017 ), did not acquire theoretical language assessment knowledge ( Mede and Atay, 2017 ; Xu and Brown, 2017 ; Kim et al, 2020 ), designed language assessment intuitively ( Sultana, 2019 ) or inappropriately interpreted students’ test results ( Kim et al, 2020 ).…”