Automatic short answer scoring using words overlapping methods

Abstract

Abstract. Assessment is one of the most important things in studying. In this digital era, there are a lot of systems that have developed to handle assessment automatically. One of the system assessments that were developed by researcher is automatic scoring for essay. There are two types of essay; long and short answer essays. This paper is focused on the development of automatic short answer scoring. Some automatic scoring systems used on long answer have shown optimal results in giving a score on the students answer. Automatic long answer systems use the information retrieval method to measure similarity between students answer and references answer. Automatic short answer scoring does not give the best result yet. Short answer has a limited word in each answer. Each answer consists of one phrase to three sentences. Assessment of the short description that has limited number of words requires special handling, especially in the weighting process. With the limitations of the process of weighting the word, it cannot be done with frequency model, because the words occurrence is very rare. This study tries to compare several methods that apply the overlapping methods to determine the degree of similarity between the references answer and students answer. From the research result shows that the method Cosine Coefficient has better results than the Dice and Jaccard Coefficient methods.

Automated Evaluation of Short Answers Using Text Similarity for the Portuguese Language

“…In general, we have found very close results considering the different measurements. These results confirm the position advocated by Pribadi et al (2017), who affirm that for short answers, set-based similarity measurements produce good results (Q3, section Method). Table 4 shows that linear regression and KNN method present similar accuracy; however, it was necessary to create one regression model for each corpus, whereas the same KNN model could be used for all distinct corpora (Q5, section Method).…”

“…In summary, were found three main approaches for assessing short answers: the first one is based on corpus and text similarities (Gomaa and Fahmy, 2012;Santos and Favero, 2015;Pribadi et al, 2017); the second one is knowledge-based similarity that rely on synonym dictionary (Benomran and Ab Aziz, 2013); the third one is corpusbased similarity that rely on a related corpus to expand the vocabulary (Mohler and Mihalcea, 2009).…”

“…These systems achieved promising results and produced accuracy levels that surpass human vs. human accuracy (Attali et al, 2010;Zupanc and Bosnic, 2017). On the other hand, automated evaluation of short answers, despite using the same techniques employed at essay scoring, has not achieved satisfactory performances (Magnini et al, 2005;Pribadi et al, 2017) and replicating decisions of a human grader for short answers remained a challenge.…”

Automatic evaluation of discursive answers short based on three linguistic dimensions / Avaliação automática de respostas discursivas curtas com base em três dimensões linguísticas

“…In the field of automatic evaluation of short discursive answers there are two main lines of research: (1) The first is based on corpus and similarity between texts (Gomaa and Fahmy 2014;Pribadi et al, 2017) and; (2) The second is based on metrics of Brazilian Journal of Development, Curitiba, v.7, n.8, p. 82824-82836 aug. 2021 similarity between networks of concepts extracted from the texts of the answers using machine learning techniques and natural language processing (PLN) (Mohler and Mihalcea 2009;Zupanc and Bosnic, 2017;Palma and Atkinson, 2018).…”

Avaliação Automática de respostas discursivas curtas baseado em três dimensões linguísticas

“…No campo da avaliação automática de questões discursivas curtas existem duas principais linhas de pesquisa: (1) A primeira baseia-se em corpus e similaridade entre textos (Gomaa e Fahmy 2014;Pribadi et al, 2017) e; (2) A segunda baseia-se em métricas de similaridade entre redes de conceitos extraídos dos textos das respostas utilizando-se técnicas de aprendizagem de máquina e processamento de linguagem natural (PLN) (Mohler e Mihalcea 2009;Zupanc e Bosnic, 2017;Palma e Atkinson, 2018).…”