Can Neural Networks Automatically Score Essay Traits?

Abstract

Essay traits are attributes of an essay that can help explain how well written (or badly written) the essay is. Examples of traits include Content, Organization, Language, Sentence Fluency, Word Choice, etc. A lot of research in the last decade has dealt with automatic holistic essay scoring -where a machine rates an essay and gives a score for the essay. However, writers need feedback, especially if they want to improve their writing -which is why traitscoring is important. In this paper, we show how a deep-learning based system can outperform feature-based machine learning systems, as well as a string kernel system in scoring essay traits.

The State of the Art in Providing Automated Feedback to Open-Ended Student Work

“…For the purposes of this review, those associated solely with the quality of the products, writing reports or openended constructed responses, such as spelling, or grammar automatic reviewers are not included. On the other hand, research on the quality of writing, such as sentence fluency, clarity, and coherence, has been researched in the past based on the classification of engineering features [45], [59]. Recent developments using deep learning methods, such as the one proposed by Mathias & Bhattacharyya [45] based on a hierarchical Convolutional Neural Network and Long Short-Term Memory (CNN-LSTM) architecture or in attention-based Recurrent Convolutional Neural Network presented by Dong et al [9], show promising results generalizing different aspects of writing quality for holistic grading and richer feedback.…”

“…Originated from the scoring point of view, automated essay scoring (AES) has a vast tradition in machine learning and natural language processing [38]. Relevant to the assessment of CPS, recent developments in AES research [15] [27], [24], [29], [45], [65], [67], [68] have been motivated by increasing attention to developing argumentation skills in STEM education to deepen scientific and critical thinking. Deep learning architectures based on LSTM with or without attention, hierarchical CNN, and the use of pre-train language models like BERT have successfully been able to extract high-order features associated with argumentation features.…”

“…In that sense, several interesting studies have been identified. Of particular interest, further research on the line of the work proposed by Gosh et al, [15], Hughes [29], Mathias and Bhattacharyya [45], Zhang and Litman [65], [67], and Zhao et al [68], will be pursued.…”

Are pre-trained text representations useful for multilingual and multi-dimensional language proficiency modeling?

“…Further, most of the approaches focused on modeling language proficiency as a single variable. Although there is some research focusing on multiple dimensions of language proficiency (Lee et al, 2009;Attali and Sinharay, 2015;Agejev and Šnajder, 2017;Mathias and Bhattacharyya, 2020), none of them focused on non-English languages or used recent multilingual pre-trained models such as BERT. In this paper, we focus on this problem of multi-dimensional modeling of language proficiency for three languages-German, Italian, and Czech-and explore whether recent research on multilingual embeddings can be useful for non-English AES.…”

Machine Learning Towards Intelligent Systems: Applications, Challenges, and Opportunities

“…Experiments showed that the accuracy of the different ML models ranged between 80%-96% for the different reflective essay metrics. Mathias and Bhattacharyya (2020) explored the use of deep learning models to automate the essay grading process. To that end, the authors used the ASAP AEG dataset that described differed essay sets with multiple essay traits such as Content, Organization, Word Choice, Sentence Fluency, and Conventions (Mathias and Bhattacharyya, 2018).…”