Coherence‐based automatic short answer scoring using sentence embedding

Abstract

Automatic essay scoring (AES) is an essential educational application in natural language processing. This automated process will alleviate the burden by increasing the reliability and consistency of the assessment. With the advances in text embedding libraries and neural network models, AES systems achieved good results in terms of accuracy. However, the actual goals still need to be attained, like embedding essays into vectors with cohesion and coherence, and providing student feedback is still challenging. In this paper, we proposed coherence‐based embedding of an essay into vectors using sentence‐Bidirectional Encoder Representation for Transformers. We trained these vectors on Long short‐term memory and bidirectional long short‐term memory to capture sentence connectivity with other sentences' semantics. We used two datasets: standard ASAP Kaggle and a domain‐specific dataset with almost 2500 responses from 650 students. Our model performed well on both datasets, with an average quadratic weighted kappa score of 0.76. Furthermore, we achieved good results compared to other prescribed models, and we also tested our model on adversarial responses of both datasets and observed decent outcomes.