Automated Scoring of Students’ Use of Text Evidence in Writing

Abstract

Despite the importance of analytic text‐based writing, relatively little is known about how to teach to this important skill. A persistent barrier to conducting research that would provide insight on best practices for teaching this form of writing is a lack of outcome measures that assess students’ analytic text‐based writing development and that are feasible to implement at scale. Automated essay‐scoring (AES) technologies offer one potential approach to increasing the feasibility of research in this area, provided that the scores yield information about substantive dimensions of writing aligned to new standards and are sensitive to variation in literacy instruction. The authors describe an approach to using AES technologies to provide information about students’ skills at marshaling text evidence in the upper elementary grades. Specifically, the authors examined 1,529 responses to a response‐to‐text assessment (RTA) from 65 fifth‐ and sixth‐grade language arts classrooms, from which the authors also collected data on instruction via logs, text‐based writing assignments, and surveys. Through correlational, univariate, and multilevel multivariate analyses, the authors found validity evidence supporting automated scoring of the RTA: The authors found close correspondence of human and AES scores, alignment of AES scores with components of instruction that the authors expected would predict variation in students’ writing quality, and association between AES scores and other expected measures of student achievement. These findings provide encouraging evidence that AES technologies as applied to the RTA can generate valid inferences about students’ ability to marshal text evidence in writing and, thus, could be a useful tool for advancing large‐scale writing research.

Application of an Automated Essay Scoring engine to English writing assessment using Many-Facet Rasch Measurement

“…The connection between the educational philosophy and practices in the use of technology are of vital importance (Andrei, 2017). Although the AES systems are able to score essays similarly to human raters (Correnti et al, 2020; Wilson, 2018), they are often accepted as scoring tools that merely complement, but cannot replace expert human raters, either in high-stakes or low-stakes assessments.…”

“…Although the scores generated by AESs are routinely close to the scores of human raters in writing assessments (Correnti et al, 2020; Rudner et al, 2006; Warschauer & Grimes, 2008), AESs are often touted as instruments to supplement, rather than to replace expert human raters. For instance, an AES engine is used by the GMAT Analytic Writing Assessment in parallel with one human rater to generate two scores for each essay.…”

HABE-IXA euskarazko idazmen-proben corpuseko idazlanen mailakatze automatikoa

“…Beraz, oso testu-bilduma handiak prozesatzeko eta analizatzeko aukera berriak daude. Hala, euskara-ikasleek ekoitzitako testuen maila neurtzeko probak egin eta zuzendu litezke hizkuntza-prozesamendurako tresnak erabiliz (Correnti et al, 2019;Maamuujav et al, 2021).…”

Computing Numerical Images of Student Writing

“…An important research area aimed at measuring the individual competence of students by using CL methods is that of writing assessment. In recent years, the application of CL methods has been particularly evident within the field of automated essay scoring, in which researchers try to develop software tools that can automatically analyze features in texts and on that basis assign similar texts with a certain score (Correnti et al, 2020;Shermis, 2014). The theoretical underpinnings of many of these studies are often located in probabilistic inference or more widely within the field of psycholinguistics (Bod, 2009;Jurafsky, 2002).…”