Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy

Abstract

The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth graders were randomly assigned by classroom to an AWE condition that used Project Essay Grade Writing ( n = 56) or a word-processing condition that used Google Docs ( n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic writing quality, performance on a state English language arts test, and teachers' perceptions of AWE's social validity. Path analyses showed that after controlling for pretest measures, composing condition had no effect on holistic writing quality, but students in the AWE condition had more positive writing self-efficacy and better performance on the state English language arts test. Posttest writing self-efficacy partially mediated the effect of composing condition on state test performance. Teachers reported positive perceptions of AWE's social validity. Results emphasize the importance of using multiple metrics and considering both contextual factors and AWE implementation methods when evaluating AWE effectiveness.

Analytical Techniques for Developing Argumentative Writing in STEM

“…Automated support for revision feedback using analytic rubrics has been applied to second language learners' persuasive essays [40], college students' physics lab reports [41], and middle school students in English language arts (ELA) classes [42]. Liu et al [40] developed machine-learned models by training on pairs of student sentences and teacher comments from a previously collected dataset of L2 learner essays.…”

“…Results indicated that teachers gave more feedback on higher-level writing skills to students in the combined condition, and that reliance on PEG saved one-third to half the time it took to provide feedback without PEG. In a somewhat larger study, middle school students in ELA classes using PEG had more positive writing self-efficacy and higher scores on the state ELA test than a control group [42]. Other machine learning methods for predicting scores using all or some of the analytic rubric dimensions have also been investigated for college level L2 essays [45] and middle school argument writing [46].…”

“…To summarize, studies show automated analysis can support formative assessment during writing instruction by helping the instructor to provide prompt feedback [40], [47], [48], to students while revising their drafts [44], [53], [55], which can lead to improved writing skills [42], [57]. Machine learning methods as used in PEG, C-rater-ML, G-Rubric and [40] generalize better than Coh-Metrix alone, although Coh-Metrix provides useful features for the machine learning approach used in [40].…”

Artificial Intelligence, Teacher Tasks and Individualized Pedagogy

“…10 This is perhaps surprising, considering 7 In particular, we are not aware of any impact evaluation that does so in a post-primary education context. Outside post-primary education, Shermis et al (2008), Palermo and Thomson (2018) and Wilson and Roscoe (2019) use experimental data on grades 6-10. However, we believe there are important limitations in the results presented in these papers.…”

“…Most outcomes in this literature are also not economically important. Wilson and Roscoe (2019) present an evaluation of the effects Project Essay Grade Writing in Texas on the state English Language Arts test but treatment was randomized using a small sample of clusters (3 teachers in 10 different classrooms) and the control group received a recommendation of using Google Docs as an alternative resource.…”