Get IT Scored Using AutoSAS — An Automated System for Scoring Short Answers

Abstract

In the era of MOOCs, online exams are taken by millions of candidates, where scoring short answers is an integral part. It becomes intractable to evaluate them by human graders. Thus, a generic automated system capable of grading these responses should be designed and deployed. In this paper, we present a fast, scalable, and accurate approach towards automated Short Answer Scoring (SAS). We propose and explain the design and development of a system for SAS, namely AutoSAS. Given a question along with its graded samples, AutoSAS can learn to grade that prompt successfully. This paper further lays down the features such as lexical diversity, Word2Vec, prompt, and content overlap that plays a pivotal role in building our proposed model. We also present a methodology for indicating the factors responsible for scoring an answer. The trained model is evaluated on an extensively used public dataset, namely Automated Student Assessment Prize Short Answer Scoring (ASAP-SAS). AutoSAS shows state-of-the-art performance and achieves better results by over 8% in some of the question prompts as measured by Quadratic Weighted Kappa (QWK), showing performance comparable to humans.

What All Do Audio Transformer Models Hear? Probing Acoustic Representations for Language Delivery and Its Structure

“…Surface-Level: The lexical diversity of spoken speech is an important metric to evaluate its quality (Read et al, 2006;Kumar et al, 2019). Surface features can be inferred by simply looking at the content spoken.…”

ESAS: Towards Practical and Explainable Short Answer Scoring (Student Abstract)

“…So, to overcome all these obstacles in the path of providing fair education, ESAS, an automated short answer scoring practical and explainable domain agnostic tool has been developed to assist teachers. The main highlight of ESAS is the extensive feedback provided along with suggestions to the students in a uniform unbaised fashion which sets it apart from the previous work (Kumar et al 2019) (Riordan et al 2017). ESAS utilizes Natural language Processing techniques (Ramachandran, Cheng, and Foltz 2015) and a few novel semantic overlap features designed to capture domain-specific knowledge and student's response similarity with sample responses.…”

Improving Automatic Essay Scoring for Indonesian Language using Simpler Model and Richer Feature

“…This could be due to the very small number of available data. Another approach is to use a tree-based model as is done in [2][3] [16] [17]. The use of tree-based models has another advantage.…”

“…The use of tree-based models has another advantage. The model can explain the scores given as shown in [16]. In the case of Indonesian AES, besides the previously mentioned methods, one of the methods that have been used is using Latent Semantic Analysis (LSA) [18] or term frequency similarity [19].…”

“…In contrast to some of the datasets used in [17] and another Indonesian AES dataset, the Indonesian Query Answering Dataset [20], which scored between 0 and maximum point. In addition, in [16] the dataset has different answer characteristics that relatively longer than the Ukara dataset.…”