Automated feedback and writing: a multi-level meta-analysis of effects on students' performance

Abstract

IntroductionAdaptive learning opportunities and individualized, timely feedback are considered to be effective support measures for students' writing in educational contexts. However, the extensive time and expertise required to analyze numerous drafts of student writing pose a barrier to teaching. Automated writing evaluation (AWE) tools can be used for individual feedback based on advances in Artificial Intelligence (AI) technology. A number of primary (quasi-)experimental studies have investigated the effect of AWE feedback on students' writing performance.MethodsThis paper provides a meta-analysis of the effectiveness of AWE feedback tools. The literature search yielded 4,462 entries, of which 20 studies (k = 84; N = 2, 828) met the pre-specified inclusion criteria. A moderator analysis investigated the impact of the characteristics of the learner, the intervention, and the outcome measures.ResultsOverall, results based on a three-level model with random effects show a medium effect (g = 0.55) of automated feedback on students' writing performance. However, the significant heterogeneity in the data indicates that the use of automated feedback tools cannot be understood as a single consistent form of intervention. Even though for some of the moderators we found substantial differences in effect sizes, none of the subgroup comparisons were statistically significant.DiscussionWe discuss these findings in light of automated feedback use in educational practice and give recommendations for future research.

Empirische Arbeit: Comparing Generative AI and Expert Feedback to Students’ Writing: Insights from Student Teachers

“…While the approach has been shown to imitate teacher judgments accurately (Horbach et al, 2022;Zhai et al, 2021) and be a suitable foun-dation for feedback systems (Fleckenstein, Liebenow & Meyer, 2023;Jansen, Meyer, Fleckenstein, Horbach, Keller & Möller, 2024), the extensive requirement of training data elevates costs and restricts teachers' flexibility in using automated feedback in the classroom. Additionally, AWE systems match pre-defined feedback sets with texts rather than generating individualized feedback contextually, which can limit their relevance and effectiveness in varying educational scenarios.…”

Predictors of middle school students’ perceptions of automated writing evaluation

“…In recent years, technology has profoundly transformed language education, particularly in the realm of writing instruction ( Wen & Walters, 2022 ), and specifically via the use of automated writing evaluation (AWE; Fleckenstein et al, 2023 ; Ngo et al, 2022 ; Nunes et al, 2022 ). Studies employing activity theory and examining AWE-supported writing processes have shed light on technology's impact on educational contexts ( Chen et al, 2022 ; Li, 2022 ).…”

“…Based on recent meta-analyses ( Fleckenstein et al, 2023 ; Graham et al, 2015 ; Li, 2022 ; Zhai & Ma, 2022b ), one promising technology-based intervention is AWE. AWE is software that uses natural language processing to provide immediate, computer-generated evaluative scores and feedback ( Hockly, 2019 ; Strobl et al, 2019 ).…”

“…AWE capabilities have expanded in recent years ( Deeva et al, 2021 ; Huang et al, 2023 ), accompanied by increased AWE adoption in schools. Along with research evaluating the effectiveness of AWE, which shows generally positive findings ( Ersanli & Yesilel, 2023 ; Fleckenstein et al, 2023 ; Li, 2022 ; Zhai & Ma, 2022b ), it also is important that research evaluate stakeholders' perceptions of the social validity of AWE (see Shi & Aryadoust, 2022 ). Social validity refers to how the goals, procedures, and effects of an intervention are perceived by its recipients and/or implementers ( Kazdin, 1977 ; Wolf, 1978 ) and may also include perceptions of an intervention's usability, usefulness, and desirability.…”