Neural Automated Essay Scoring Incorporating Handcrafted Features

Abstract

Automated essay scoring (AES) is the task of automatically assigning scores to essays as an alternative to grading by human raters. Conventional AES typically relies on handcrafted features, whereas recent studies have proposed AES models based on deep neural networks (DNNs) to obviate the need for feature engineering. Furthermore, hybrid methods that integrate handcrafted features in a DNN-AES model have been recently developed and have achieved state-of-the-art accuracy. One of the most popular hybrid methods is formulated as a DNN-AES model with an additional recurrent neural network (RNN) that processes a sequence of handcrafted sentencelevel features. However, this method has the following problems: 1) It cannot incorporate effective essay-level features developed in previous AES research. 2) It greatly increases the numbers of model parameters and tuning parameters, increasing the difficulty of model training. 3) It has an additional RNN to process sentence-level features, enabling extension to various DNN-AES models complex. To resolve these problems, we propose a new hybrid method that integrates handcrafted essay-level features into a DNN-AES model. Specifically, our method concatenates handcrafted essay-level features to a distributed essay representation vector, which is obtained from an intermediate layer of a DNN-AES model. Our method is a simple DNN-AES extension, but significantly improves scoring accuracy.

Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

“…Additionally, our model could be enhanced by multiple techniques found in other works such as adding domain-specific features [47,51] or recognizing question types or their difficulty [34], and utilizing specialized methods for each of them though on the downside this increases system complexity. It is especially interesting to leverage rubrics as defined by teachers.…”

Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays

“…Most modern NLP systems have started to use attention-based transformer networks and large pretrained language models. Yang et al (2020), Uto et al (2020) use the We have uploaded the rest of the requirements along with the code base-uncased (Devlin et al, 2019) pre-trained language model to perofrm automatic essay grading achieving QWKs in the range of 0.79 to 0.805. However, BERT has about 110 million parameters (compared to our largest model with just under 2 million parameters).…”

UMass PCL at SemEval-2022 Task 4: Pre-trained Language Model Ensembles for Detecting Patronizing and Condescending Language

“…Transformers are generally able to vastly outperform regression on engineered features. However, in some text labeling tasks, such as essay scoring, it has been shown that engineered features can be used in tandem with Transformer output to improve performance (Uto et al, 2020). This can be achieved simply by concatenating a vector of features f n to BERT's CLS vector.…”

“…Our set of features was inspired by (Uto et al, 2020), but we excluded the readability metrics because they are not as relevant for our task. Specifically, for text sample x n , we calculate the number of words, number of sentences, number of exclamation marks, question marks, and commas, average word length, average sentence length, the number of nouns, verbs, adjectives, and adverbs, and the number of stop words.…”