Automated Essay Scoring Using Transformer Models

Abstract

Automated essay scoring (AES) is gaining increasing attention in the education sector as it significantly reduces the burden of manual scoring and allows ad hoc feedback for learners. Natural language processing based on machine learning has been shown to be particularly suitable for text classification and AES. While many machine-learning approaches for AES still rely on a bag of words (BOW) approach, we consider a transformer-based approach in this paper, compare its performance to a logistic regression model based on the BOW approach, and discuss their differences. The analysis is based on 2088 email responses to a problem-solving task that were manually labeled in terms of politeness. Both transformer models considered in the analysis outperformed without any hyperparameter tuning of the regression-based model. We argue that, for AES tasks such as politeness classification, the transformer-based approach has significant advantages, while a BOW approach suffers from not taking word order into account and reducing the words to their stem. Further, we show how such models can help increase the accuracy of human raters, and we provide a detailed instruction on how to implement transformer-based models for one’s own purposes.

Editorial of the Psych Special Issue “Computational Aspects, Statistical Algorithms and Software in Psychometrics”

“…Ludwig et al [29] apply a transformer-based approach to automated essay scoring in the Python software and compared it with the bag of words approach. The authors argue that the transformer-based approach has significant advantages, while a bag of words approach suffers from not taking word order into account and reducing the words to their stem.…”

Effect of Feedback and Strategy Training on Undergraduate Students’ Writing Ability

“…Furthermore, a variety of feedback sources may be most useful for providing both error correction and motivation to practise and improve. Nonetheless, essay scoring automation has the potential to significantly increase the amount of writing assigned to students and the rate at which they receive feedback [22,23].…”

Digital Distance Learning and the Transformation of Vocational Schools From a Qualitative Perspective

“…Additionally, digital tools can help teachers to enhance feedback culture in the classroom with positive implications for their workload. Digital quizzes can be implemented to give direct feedback to the students (George, 2020), as well as automated scoring for longer texts (Ludwig et al, 2021). On the other hand, students have to be aware of the fact that they might not receive individual feedback for each assignment and they have to further develop competencies to evaluate their learning process.…”