The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis

Abstract

Automated writing evaluation (AWE) has been frequently used to provide feedback on student writing. Many empirical studies have examined the effectiveness of AWE on writing quality, but the results were inconclusive. Thus, the magnitude of AWE’s overall effect and factors influencing its effectiveness across studies remained unclear. This study re-examined the issue by meta-analyzing the results of 26 primary studies with a total of 2468 participants from 2010 to 2022. The results revealed that AWE had a large positive overall effect on writing quality (g = 0.861, p < 0.001). Further moderator analyses indicated that AWE was more effective for post-secondary students than for secondary students and had more benefits for English as a Foreign Language (EFL) and English as a Second Language (ESL) learners than for Native English Speaker (NES) learners. When the genre of writing was considered, AWE showed a more significant impact on argumentative writing than on academic and mixed writing genres. However, intervention duration, feedback combination, and AWE platform did not moderate the effect of AWE on writing quality. The implications and recommendations for both research and practice are discussed in depth.

Automated feedback and writing: a multi-level meta-analysis of effects on students' performance

“…We differentiated for secondary and tertiary level, finding similar effects of medium size for both. This is contrary to the findings by Ngo et al ( 2022 ) and Zhai and Ma ( 2022 ), who both found larger effects for post-secondary learners compared to learners at secondary level. However, both previous meta-analyses only included a very limited number of primary studies drawing on secondary-level samples ( k = 3 resp.…”

“…This should be kept in mind when verbally classifying the effect sizes. In the following, we interpret our findings in light of previous research, especially the two recent meta-analyses by Ngo et al ( 2022 ) and Zhai and Ma ( 2022 ).…”

“…In terms of language context, the effect was large for L2 and medium for L1 contexts. Zhai and Ma ( 2022 ) reported a similar finding when comparing learners of English as a second or foreign language with native English speakers.…”

Predictors of middle school students’ perceptions of automated writing evaluation

“…Based on recent meta-analyses ( Fleckenstein et al, 2023 ; Graham et al, 2015 ; Li, 2022 ; Zhai & Ma, 2022b ), one promising technology-based intervention is AWE. AWE is software that uses natural language processing to provide immediate, computer-generated evaluative scores and feedback ( Hockly, 2019 ; Strobl et al, 2019 ).…”

“…AWE capabilities have expanded in recent years ( Deeva et al, 2021 ; Huang et al, 2023 ), accompanied by increased AWE adoption in schools. Along with research evaluating the effectiveness of AWE, which shows generally positive findings ( Ersanli & Yesilel, 2023 ; Fleckenstein et al, 2023 ; Li, 2022 ; Zhai & Ma, 2022b ), it also is important that research evaluate stakeholders' perceptions of the social validity of AWE (see Shi & Aryadoust, 2022 ). Social validity refers to how the goals, procedures, and effects of an intervention are perceived by its recipients and/or implementers ( Kazdin, 1977 ; Wolf, 1978 ) and may also include perceptions of an intervention's usability, usefulness, and desirability.…”