Bringing Automatic Scoring into the Classroom – Measuring the Impact of Automated Analytic Feedback on Student Writing Performance

Abstract

While many methods for automatically scoring student writings have been proposed, few studies have inquired whether such scores constitute effective feedback improving learners’ writing quality. In this paper, we use an EFL email dataset annotated according to five analytic assessment criteria to train a classifier for each criterion, reaching human-machine agreement values (kappa) between .35 and .87. We then perform an intervention study with 112 lower secondary students in which participants in the feedback condition received stepwise automatic feedback for each criterion while students in the control group received only a description of the respective scoring criterion. We manually and automatically score the resulting revisions to measure the effect of automated feedback and find that students in the feedback condition improved more than in the control group for 2 out of 5 criteria. Our results are encouraging as they show that even imperfect automated feedback can be successfully used in the classroom.

Empirische Arbeit: Comparing Generative AI and Expert Feedback to Students’ Writing: Insights from Student Teachers

“…While the approach has been shown to imitate teacher judgments accurately (Horbach et al, 2022;Zhai et al, 2021) and be a suitable foun-dation for feedback systems (Fleckenstein, Liebenow & Meyer, 2023;Jansen, Meyer, Fleckenstein, Horbach, Keller & Möller, 2024), the extensive requirement of training data elevates costs and restricts teachers' flexibility in using automated feedback in the classroom. Additionally, AWE systems match pre-defined feedback sets with texts rather than generating individualized feedback contextually, which can limit their relevance and effectiveness in varying educational scenarios.…”