On the Effectiveness of Curriculum Learning in Educational Text Scoring

Abstract

Automatic Text Scoring (ATS) is a widely-investigated task in education. Existing approaches often stressed the structure design of an ATS model and neglected the training process of the model. Considering the difficult nature of this task, we argued that the performance of an ATS model could be potentially boosted by carefully selecting data of varying complexities in the training process. Therefore, we aimed to investigate the effectiveness of curriculum learning (CL) in scoring educational text. Specifically, we designed two types of difficulty measurers: (i) pre-defined, calculated by measuring a sample's readability, length, the number of grammatical errors or unique words it contains; and (ii) automatic, calculated based on whether a model in a training epoch can accurately score the samples. These measurers were tested in both the easy-to-hard to hard-to-easy training paradigms. Through extensive evaluations on two widely-used datasets (one for short answer scoring and the other for long essay scoring), we demonstrated that (a) CL indeed could boost the performance of state-of-the-art ATS models, and the maximum improvement could be up to 4.5%, but most improvements were achieved when assessing short and easy answers; (b) the pre-defined measurer calculated based on the number of grammatical errors contained in a text sample tended to outperform the other difficulty measurers across different training paradigms.

Practical and ethical challenges of large language models in education: A systematic scoping review

“…Although educational research that leverages LLMs to develop technological innovations for automating educational tasks is yet to achieve its full potential (ie, most works have focused on improving model performances (Kurdi et al, 2020;Ramesh & Sanampudi, 2022)), a growing body of literature hints at how different stakeholders could potentially benefit from such innovations. Specifically, these innovations could potentially play a vital role in addressing teachers' high levels of stress and burnout by reducing their heavy workloads by automating punctual, time-consuming tasks (Carroll et al, 2022) such as question generation (Bulut & Yildirim-Erbasli, 2022;Kurdi et al, 2020;Oleny, 2023), feedback provision (Cavalcanti et al, 2021;Nye et al, 2023), scoring essays (Ramesh & Sanampudi, 2022) and short answers (Zeng et al, 2023). These innovations could also potentially benefit both students and institutions by improving the efficiency of often tedious administrative processes such as learning resource recommendation, course recommendation and student feedback evaluation, potentially (Sridhar et al, 2023;Wollny et al, 2021;Zawacki-Richter et al, 2019).…”