Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking

Abstract

Automated Essay Scoring (AES) is a critical text regression task that automatically assigns scores to essays based on their writing quality. Recently, the performance of sentence prediction tasks has been largely improved by using Pre-trained Language Models via fusing representations from different layers, constructing an auxiliary sentence, using multitask learning, etc. However, to solve the AES task, previous works utilize shallow neural networks to learn essay representations and constrain calculated scores with regression loss or ranking loss, respectively. Since shallow neural networks trained on limited samples show poor performance to capture deep semantic of texts. And without an accurate scoring function, ranking loss and regression loss measures two different aspects of the calculated scores. To improve AES's performance, we find a new way to fine-tune pre-trained language models with multiple losses of the same task. In this paper, we propose to utilize a pretrained language model to learn text representations first. With scores calculated from the representations, mean square error loss and the batch-wise ListNet loss with dynamic weights constrain the scores simultaneously. We utilize Quadratic Weighted Kappa to evaluate our model on the Automated Student Assessment Prize dataset. Our model outperforms not only state-of-the-art neural models near 3 percent but also the latest statistic model. Especially on the two narrative prompts, our model performs much better than all other state-of-theart models.

Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

“…This idea might also be added to our approach. A neural network-based approach to fine-tune a BERT model is discussed in [54,56]. [31] states that pre-trained models may gen-eralize better on new essay topics that the trained model has not seen yet than simple neural networks such as LSTMs.…”

A review of deep-neural automated essay scoring models

“…Then, the embedding representation w t corresponding to w t is calculable as a dot product w t = A ⋅ w t . (Taghipour and Ng 2016;Alikaniotis et al 2016) -Hierarchical representation models (Dong and Zhang 2016;Dong et al 2017), -Coherence models (Tay et al 2018;Li et al 2018;Farag et al 2018;Mesgar and Strube 2018;Yang and Zhong 2021), -BERT-based models (Nadeem et al 2019;Rodriguez et al 2019;Yang et al 2020;Mayfield and Black 2020), -Hybrid models (Dasgupta et al 2018; -Robust model (Uto and Okano 2020)…”

“…5. Furthermore, Yang et al (2020) proposed fine-tuning the BERT model so that the essay scoring task and an essay ranking task are jointly resolved. As shown in Fig.…”

“…The study by Song et al (2020) reported that incorporating the above two-stage pre-training and fine-tuning improves the performance of the target-prompt scoring. Cao et al (2020) proposed another cross-prompt holistic scoring model that was designed to solve the AES task with two prompt-independent self-supervised learning tasks jointly. The two self-supervised learning tasks, which are appended to efficiently extract prompt-independent common knowledge, are a sentence reordering task and a noise identification task, as explained bellow.…”

“…Furthermore, Cao et al (2020) proposed a technique to improve the adaptability of the model to a target prompt. Concretely, during the model training processes, ( 14)…”