Using the Developmental Path of Cause to Bridge the Gap between AWE Scores and Writing Teachers’ Evaluations

Abstract

Supported by artificial intelligence (AI), the most advanced Automatic Writing Evaluation (AWE) systems have gained increasing attention for their ability to provide immediate scoring and formative feedback, yet teachers have been hesitant to implement them into their classes because correlations between the grades they assign and the AWE scores have generally been low. This begs the question of where improvements in evaluation may need to be made, and what approaches are available to carry out this improvement. This mixed-method study involved 59 cause and effect essays collected from English language learners enrolled in six different sections of a college level academic writing course and utilized theory proposed by Slater and Mohan (2010) regarding the developmental path of cause. The study compared the results of raters who used this developmental path with the accuracy of AWE scores produced by Criterion, an AWE tool developed by Educational Testing Service (ETS), and the grades reported by teachers. Findings suggested that if Criterion is to be used successfully in the classroom, writing teachers need to take a meaning-based approach to their assessment, which would allow them and their students to understand more fully how language constructs cause and effect. Using the developmental path of cause as an analytical framework for assessment may then help teachers assign grades that are more in sync with AWE scores, which in turn can help students gain more trust in the scores they receive from both their teachers and Criterion. Disciplines Educational Assessment, Evaluation, and Research | Language and Literacy Education Comments This is a manuscript of an article published as Ma, Hong, and Tammy Slater. "Using the Developmental Path of Cause to Bridge the Gap between AWE Scores and Writing Teachers' Evaluations." Writing & Pedagogy 7 (2015). 10.1558/wap.v7i2-3.26376 Posted with permission.

Systematic review of research on artificial intelligence applications in higher education – where are the educators?

“…However, they stressed that not all writing genres may be appropriate for AES and that it would be impractical to use in most small classrooms, due to the need to calibrate the system with a large number of pre-scored assessments. The benefits of using algorithms that find patterns in text responses, however, has been found to lead to encouraging more revisions by students (Ma & Slater, 2015) and to move away from merely measuring student knowledge and abilities by multiple choice tests (Nehm, Ha, & Mayfield, 2012). Continuing issues persist, however, in the quality of feedback provided by AES (Dikli, 2010), with Barker (2011) finding that the more detailed the feedback provided was, the more likely students were to question their grades, and a question was raised over the benefits of this feedback for beginning language students (Aluthman, 2016).…”

ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?

“…Automated Essay Scoring (AES) systems are the most common AI-powered assessments and can be applied across various disciplines, but most of the research has focused on its application to undergraduate courses (Zawacki-Richter et al, 2019). There are a variety of methods of developing AES systems, such as statistical modelling, natural language processing (NLP) and Latent Semantic Analysis (LSA), and the algorithms can be used to identify patterns in text responses and prompt students to revise their responses (Ma & Slater, 2015). This, in turn, could allow educators to consider a broader range of assessment methods than only using multiple-choice tests to assess students' knowledge and abilities.…”

Cognitive Engagement of Low Proficiency Learners: Evaluating the Impact of ChatGPT Feedback in EFL Writing

“…Despite two studies reporting no statistically significant change in academic performance, both Topal et al (2021) and Yang & Shulruf (2019) observed increased learning interest and confidence among students. While AI can be an invaluable tool in the assessment and feedback process, it is not without its limitations, particularly in comprehending intricate subject matter, contextual nuances, and the distinctive qualities of individual students' work [4] [5] . Moreover, there is a lack of standardized criteria for evaluating the appropriateness of grading systems across diverse contexts [6] .…”