KASS: Korean Automatic Scoring System for Short-answer Questions

Abstract

The scoring of short-answer questions in a national-wide achievement test to public school students needs a lot of human efforts and financial expenses. Since we know that natural language processing technology can be applied to replace the manual scoring process by automatic scoring software, many researchers tried to build an automatic scoring system like crater and e-rater in English. In this paper, we explored a Korean automatic scoring system for short and free-text responses. NLP techniques like morphological analysis are used to build a token-based scoring template for increasing the coverage of the automatic scoring process. We performed an experiment to measure the efficiency of the automatic scoring system and it covered about 90 to 95% of the student responses with an agreement rate 95% to the manual scoring.

How Reliable Is It to Automatically Score Open-Ended Items? An Application in the Turkish Language

“…In this research, three algorithms based on classical machine learning (logistic regression [LR], multinominal Naive Bayes [MNB], support vector machines [SVM]) and two deep learning algorithms based on artificial neural networks (longshort term memory [LSTM], bidirectional long-short term memory [BLSTM]) were used. Detailed information about these algorithms can be found in Berg and Gopinathan (2017), Gierl et al (2014), Jang, Kang, Noh, Kim, Sung, and Seong (2014), and Lilja (2018).…”

“…The differentiation of these features from other language families requires reviewing automated scoring studies in the Altaic language family. Jang et al (2014) conducted research on the Korean language and Ishioka and Kameda (2006) on the Japanese language. In the two studies mentioned, algorithms in which properties are defined manually were used.…”

“…Although automated scoring systems developed in languages that have similar features to the Turkish language are not based on supervised machine learning, they can be used similarly. Ishioka and Kameda (2006) and Jang et al (2014) determined that there was a high level of correlation between the automated scoring system and human scores in the Japanese language and the Korean language, respectively.…”

“…The automated scoring system created in the Turkish language can be used in large-scale tests. It was also stated that the automated scoring system created in Korean, which is a similar language to Turkish, can be used in large-scale tests (Jang et al, 2014). Based on the findings obtained as a result of the research, the recommendations for researchers and practitioners are as follows:…”