Coherence based automatic short answer scoring using sentence Embedding

Abstract

Automatic essay scoring is an essential educational application in natural language processing (NLP). This automated process will alleviate the burden and increase the reliability and consistency of the assessment. With the advance in text embedding libraries and neural network models, AES systems achieved good results in terms of accuracy. However, the actual goals are not attained, like embedding essays into vectors with cohesion and coherence, and providing feedback to students is still challenging. In this paper, we proposed coherence-based embedding of an essay into vectors using sentence-BERT (Bidirectional Encoder Representations from Transformers). We trained these vectors on Long Short-Term Memory (LSTM) and Bi-LSTM (Bidirectional Long Short-Term Memory) to capture sentence connectivity with other sentences' semantics. We used two different datasets; one is standard ASAP Kaggle, and another is a domain-specific dataset with almost 2500 responses from 650 students. Our model performed well on both datasets, with an average QWK (Quadratic Weighted Kappa ) score of 0.76. Furthermore, we achieved good results compared to other prescribed models, and we also tested our model on adversarial responses of both datasets and observed decent outcomes.