Unveiling the Scoring Validity of Two Chinese Automated Writing Evaluation Systems: A Quantitative Study

Abstract

Computer Assisted Language Learning (CALL) has been a burgeoning industry in China, one case in point being the extensive employment of Automated Writing Evaluation (AWE) systems in college English writing instruction to reduce teachers’ workload. Nonetheless, what warrants a special mention is that most teachers include automatic scores in the formative evaluation of relevant courses with scant attention to the scoring efficacy of these systems (Bai & Wang, 2018; Wang & Zhang, 2020). To have a clearer picture of the scoring validity of two commercially available Chinese AWE systems (Pigai and iWrite), the present study sampled 486 timed CET-4 (College English Test Band-4) essays produced by second-year non-English majors from 8 intact classes. Data comprising the maximum score difference, agreement rate, Pearson’s correlation coefficient and Cohen’s Kappa were collected to showcase human-machine and machine-machine congruence. Quantitative linguistic features of the sample essays, including accuracy, lexical and syntactic complexity, and discourse features, were also gleaned to investigate the differences (or similarities) in construct representation valued by both systems and human raters. Results show that (1) Pigai and iWrite largely agreed with each other but differed a lot from human raters in essay scoring; (2) high-human-score essays were prone to be assigned low machine scores; (3) machines relied heavily on the quantifiable features, which, however, had limited impacts on human raters.

Research Progress and Trends of Pigai.org Automated Writing Evaluation System in English Writing: A Systematic Bibliometric Analysis (2011-2023)

“…(Zhang, 2020). As Pigai.org AWE system can produce an overall essay score by estimating the quantitative differences between the essay and the texts in its corpus in four areas: vocabulary, sentence structure, organization and structure, and subject relevance (Wang & Bai, 2021), it is therefore, students can submit and resubmit their drafts following the comments. The amount of time and effort students put into their writing by the number of revisions they made could be measured with Pigai.org and checked by teachers.…”

“…In terms of the research on Pigai.org in English Writing, researchers have mainly focused on the introduction, application and suggestions for effective collaboration use (Zhu, 2019;Luo & Liang, 2022& Zhang, 2022 or on its application in English writing to construct a multiple feedback writing mechanism (Liu, 2022;Yao, 2021;Wang & Bai, 2021), or case studies in one particular university, including a combined teaching module, an assessment and evaluation method, or a tentative study on its validity ( Tan, 2019;Yan, 2019;Guo, 2020&Yang et al, 2022. The researchers are mainly from educational backgrounds, but their research focuses vary from each other.…”

Algorithmic Translation Correction Mechanisms: An End-to-end Algorithmic Implementation of English-Chinese Machine Translation

“…In the 1950s and 1970s, researchers believed that natural language processing was a cognitive process of language, so natural language processing mainly used a rule-based approach. The rule-based approach was implemented by programmers and linguists manually writing translation rules, but it had apparent drawbacks [8]. First, it is only possible to create rules for some proposals.…”