Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction

Li, Jinrong; Link, Stephanie; Hegelheimer, Volker

doi:10.1016/j.jslw.2014.10.004

Cited by 190 publications

(137 citation statements)

References 28 publications

Supporting

Mentioning

125

Contrasting

Order By: Relevance

“…Despite problems with the accuracy of the feedback, participants from the lower-level course were able to take better advantage of the feedback in correcting errors, probably because their course's more explicit attention to linguistic form seemed to predispose them to making better use of corrective feedback. This suggests that the use of Criterion may be more justified on courses where there is a congruent focus on form, supporting previous research showing that the manner in which AWE is integrated into instruction influences its acceptance by students (Chen & Cheng, 2008;Li, Link, & Hegelheimer, 2015). We see this distinction applying to the lower-and higher-level courses in our program, and for this reason, we determined that use of Criterion was more justifiable in the lower-level course.…”

Section: Discussionsupporting

confidence: 85%

“…Given that L2 learners have a greater need for feedback on sentence-level correctness, which AWE systems are more computationally adept at providing compared to feedback on higher-level concerns (Weigle, 2013a), a case can be made for the use of AWE as a complement to instructor feedback in L2 writing classrooms. In this role, AWE promises greater autonomy for students while potentially freeing up instructors to devote their feedback efforts to aspects of writing that require human evaluation (Chen & Cheng, 2008;Li, Link, & Hegelheimer, 2015;Warschauer & Grimes, 2008) such as audience awareness and communicative effectiveness.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

Ranalli

Link

Chukharev‐Hudilainen

2016

Educational Psychology

Self Cite

100

View full text Add to dashboard Cite

An increasing number of studies on the use of tools for automated writing evaluation (AWE) in writing classrooms suggest growing interest in their potential for formative assessment. As with all assessments, these applications should be validated in terms of their intended interpretations and uses. A recent argument-based validation framework outlined inferences that require backing to support integration of one AWE tool, Criterion, into a college-level English as a Second Language (ESL) writing course. The present research appraised evidence for the assumptions underlying two inferences in this argument. In the first of two studies, we assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback. The second study focused on the utilisation inference involving the assumption that Criterion feedback is useful for students to make decisions about revisions. Results showed accuracy varied considerably across error types, as did students' abilities to use Criterion feedback to correct written errors. The findings can inform discussion of whether and how to integrate the use of AWE into writing classrooms while raising important questions regarding standards for validation of AWE as formative assessment, Criterion developers' approach to accuracy, and instructors' assumptions about the underlying purposes of AWE-based writing activities. KeywordsAcademic writing, argument-based validation, automated writing evaluation, ESL, formative assessment AWE FOR FORMATIVE ASSESSMENT 2 AbstractAn increasing number of studies on the use of tools for automated writing evaluation (AWE) in writing classrooms suggests growing interest in their potential for formative assessment. As with all assessments, these applications should be validated in terms of their intended interpretations and uses (Kane, 2012). A recent argument-based validation framework outlined inferences that require backing to support integration of one AWE tool, Criterion, into a college-level ESL writing course. The present research appraised evidence for the assumptions underlying two inferences in this argument. In the first of two studies, we assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback. The second study focused on the utilization inference involving the assumption that Criterion feedback is useful for students to make decisions about revisions. Results showed accuracy varied considerably across error types, as did students' abilities to use Criterion feedback to correct written errors. The findings can inform discussion of whether and how to integrate the use of AWE into writing classrooms while raising important questions regarding standards for validation of AWE as formative assessment, Criterion developers' approach to accuracy, and instructors' assumptions about the underlying purposes of AWE-based writing activities.

show abstract

Section: Discussionsupporting

confidence: 85%

Section: Introductionmentioning

confidence: 99%

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

Ranalli

Link

Chukharev‐Hudilainen

2016

Educational Psychology

Self Cite

100

View full text Add to dashboard Cite

show abstract

“…Some teachers, however, employ AWE to complement more germane types of activities such as pre-writing, writing practice, peer review, revision, and teacher commenting. The outcomes in these cases can be more satisfactory both for students and teachers (Li, Link, & Hegelheimer, 2015;. Although teachers do not seem to dislike AWE, they may rarely use it in their classrooms, especially because some writing tasks are representative of genres outside the software's analytical capabilities.…”

Section: Washbackmentioning

confidence: 99%

“…From this perspective, Chapelle et al (2015) provide evidence supporting the assumption that L2 students use automated feedback to make decisions on how to revise their drafts and correct errors. They also suggest a rebuttal, stating that students may not make sufficient or effective use of feedback because they may lack confidence in its accuracy (a theme also mentioned by Li et al, 2015). More broadly, interpretive arguments for AWE systems could include formulating and testing both claims based on principles guiding classroom praxis and potential rebuttals in terms of anticipated negative effects.…”

Section: Need For Empirical Evidencementioning

confidence: 99%

Automated Writing Analysis for Writing Pedagogy

Cotos

2015

WAP

View full text Add to dashboard Cite

This article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical evidence that would corroborate or refute interpretations, uses, and consequences of automated scoring and feedback tools intended for specific contexts. Such evidence can be obtained by adopting a validity argument framework. To demonstrate an application of this framework, the article presents a novel genre-based approach to automated analysis configured to support research writing and provides examples of validity evidence for using it with novice scholarly writers. Automated Writing Analysis for writing pedagogy: From healthy tension to tangible prospects Elena Cotos AbstractThis article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical evidence that would corroborate or refute interpretations, uses, and consequences of automated scoring and feedback tools intended for specific contexts. Such evidence can be obtained by adopting a validity argument framework. To demonstrate an application of this framework, the article presents a novel genre-based approach to automated analysis configured to support research writing and provides examples of validity evidence for using it with novice scholarly writers.

show abstract

“…Ohta (2008a), Hosogoshi et al (2012), and Tajino et al (2011) used Criterion® holistic scores, and Ohta (2008aOhta ( , 2008b, Hosogoshi et al (2012), and Li et al (2015) used writing quality measures derived from Criterion® Trait Feedback. All the studies mentioned above reported significant improvement in Criterion® holistic and other writing quality scores.…”

Section: Introductionmentioning

confidence: 99%

Validity evidence of Criterion® for assessing L2 writing proficiency in a Japanese university context

et al. 2016

View full text Add to dashboard Cite

Background: While numerous articles on Criterion® have been published and its validity evidence has accumulated, test users need to obtain relevant validity evidence for their local context and develop their own validity argument. This paper aims to provide validity evidence for the interpretation and use of Criterion® for assessing second language (L2) writing proficiency at a university in Japan. Method: We focused on three perspectives: (a) differences in the difficulty of prompts in terms of Criterion® holistic scores, (b) relationships between Criterion® holistic scores and indicators of L2 proficiency, and (c) changes in Criterion® holistic and writing quality scores at three time points over 28 weeks. We used Rasch analysis (to examine (a)), Pearson product-moment correlations (to examine (b)), and multilevel modeling (to examine (c)). Results: First, we found statistically significant but minor differences in prompt difficulty. Second, Criterion® holistic scores were found to be relatively weakly but positively correlated with indicators of L2 proficiency. Third, Criterion® holistic and writing quality scores-particularly, essay length and syntactic complexity-significantly improved, and thus are sensitive measures of the longitudinal development of L2 writing.

show abstract

Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction

Cited by 190 publications

References 28 publications

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation

Automated Writing Analysis for Writing Pedagogy

Validity evidence of Criterion® for assessing L2 writing proficiency in a Japanese university context

Contact Info

Product

Resources

About