Preventing Critical Scoring Errors in Short Answer Scoring with Confidence Estimation

Abstract

Many recent Short Answer Scoring (SAS) systems have employed Quadratic Weighted Kappa (QWK) as the evaluation measure of their systems. However, we hypothesize that QWK is unsatisfactory for the evaluation of the SAS systems when we consider measuring their effectiveness in actual usage. We introduce a new task formulation of SAS that matches the actual usage. In our formulation, the SAS systems should extract as many scoring predictions that are not critical scoring errors (CSEs). We conduct the experiments in our new task formulation and demonstrate that a typical SAS system can predict scores with zero CSE for approximately 50% of test data at maximum by filtering out low-reliablility predictions on the basis of a certain confidence estimation. This result directly indicates the possibility of reducing half the scoring cost of human raters, which is more preferable for the evaluation of SAS systems.

Balancing Cost and Quality: An Exploration of Human-in-the-Loop Frameworks for Automated Short Answer Scoring

“…For automatic scoring coverage, the posterior tends to be slightly dominant in ASAP-SAS, whereas the trust score is slightly dominant in the RIKEN dataset. The slightly higher performance of the trust score for the RIKEN dataset is consistent with Funayama et al [5], which was the only study, to the best of our knowledge, to utilize confidence in the SAS field.…”

“…In this study, we extend the work of Funayma et al [5], and propose a new framework for minimizing human scoring costs while controlling the overall scoring quality of the combining human scoring and automated scoring. We also conducted cross-lingual experiments using a Japanese SAS dataset, as well as the ASAP dataset commonly used in the SAS field.…”

“…Trust score. Funayama et al [5] used the trust score [10] to estimate confidence in automatic scoring of written answers. Here, we also consider using the trust score as one of the confidence estimation methods.…”

“…To the best of our knowledge, the only study in which the use of confidence estimation in SAS was investigated by Funayama et al [5]. They introduced the concept of unacceptable critical scoring errors (CSEs).…”

Data Augmentation by Rubrics for Short Answer Grading

“…SAG is the task of estimating scores of short-text answers written as an answer to a given prompt, on the basis of whether the answer satisfies the rubrics prepared by a human in advance (Mohler et al 2011;Funayama et al 2020;Mizumoto et al 2019). The SAG systems play a central role in providing stable and sustainable scoring in repeated and large-scale examinations and (online) self-study learning support systems (Attali and Burstein 2006;Mizumoto et al 2019; Shermis et al 2010;Leacock and Chodorow 2003;Burrows et al 2015).…”