Evaluating Semantic Analysis Methods for Short Answer Grading Using Linear Regression

Nau, Jonathan; Filho, Aluízio Haendchen; Passero, Guilherme

doi:10.20319/pijss.2017.32.437450

Cited by 4 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Broadly speaking, most of the existing ATS can be categorized into two categories (Bonthu, Rama Sree, and Krishna Prasad 2021; Uto, Xie, and Ueno 2020). One is often built upon traditional machine learning techniques e.g., Linear Regression (Nau, Haendchen Filho, and Passero 2017), Support Vector Machine (Gleize and Grau 2013), and Random Forests (Ishioka and Kameda 2017), whose performance is heavily dependent on the availability and quality of hand-crafted features such as the number of words contained in an answer (Platanios et al 2019) and the number of distinct words in the answer (Li et al 2016). The other is empowered by the recent deep learning techniques such as Bi-LSTM (Kim, Vizitei, and Ganapathi 2018) and BERT (Sung, Dhamecha, and Mukhi 2019), which can directly transform the raw text input as embedding-based representations to generate an assessment score without the need of manual feature engineering.…”

Section: Related Work Automatic Text Scoring In Educationmentioning

confidence: 99%

On the Effectiveness of Curriculum Learning in Educational Text Scoring

Zeng

Gašević

Chen

2023

AAAI

View full text Add to dashboard Cite

Automatic Text Scoring (ATS) is a widely-investigated task in education. Existing approaches often stressed the structure design of an ATS model and neglected the training process of the model. Considering the difficult nature of this task, we argued that the performance of an ATS model could be potentially boosted by carefully selecting data of varying complexities in the training process. Therefore, we aimed to investigate the effectiveness of curriculum learning (CL) in scoring educational text. Specifically, we designed two types of difficulty measurers: (i) pre-defined, calculated by measuring a sample's readability, length, the number of grammatical errors or unique words it contains; and (ii) automatic, calculated based on whether a model in a training epoch can accurately score the samples. These measurers were tested in both the easy-to-hard to hard-to-easy training paradigms. Through extensive evaluations on two widely-used datasets (one for short answer scoring and the other for long essay scoring), we demonstrated that (a) CL indeed could boost the performance of state-of-the-art ATS models, and the maximum improvement could be up to 4.5%, but most improvements were achieved when assessing short and easy answers; (b) the pre-defined measurer calculated based on the number of grammatical errors contained in a text sample tended to outperform the other difficulty measurers across different training paradigms.

show abstract

Section: Related Work Automatic Text Scoring In Educationmentioning

confidence: 99%

On the Effectiveness of Curriculum Learning in Educational Text Scoring

Zeng

Gašević

Chen

2023

AAAI

View full text Add to dashboard Cite

show abstract

“…As surveyed in (Bonthu et al, 2021), the approaches used to tackle ASAS often fall into two categories. One is based on traditional machine learning techniques such as SVM (Gleize and Grau, 2013;Mohler et al, 2011;Higgins et al, 2014), K-means (Sorour et al, 2015, Linear Regression (Nau et al, 2017;Heilman and Madnani, 2015;Higgins et al, 2014), and Random Forests (Higgins et al, 2014;Ramachandran et al, 2015;Ishioka and Kameda, 2017), all of which heavily rely on the input of manually-crafted features. For example, Sultan et al ( 2016) devised a set of features which were based on a lexical similarity (i.e., similarities between words identified by a paraphrase database (Ganitkevitch et al, 2013)) and monolingual alignment (Sultan et al, 2014), and input the designed features to a ridge regression model to obtain the score of an answer.…”

Section: Automated Short Answer Scoringmentioning

confidence: 99%

Do Deep Neural Nets Display Human-like Attention in Short Answer Scoring?

Zeng¹,

Li²,

Gašević³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Deep Learning (DL) techniques have been increasingly adopted for Automatic Text Scoring in education. However, these techniques often suffer from their inabilities to explain and justify how a prediction is made, which, unavoidably, decreases their trustworthiness and hinders educators from embracing them in practice. This study aimed to investigate whether (and to what extent) DL-based graders align with human graders regarding the important words they identify when marking short answer questions. To this end, we first conducted a user study to ask human graders to manually annotate important words in assessing answer quality and then measured the overlap between these human-annotated words and those identified by DL-based graders (i.e., those receiving large attention weights). Furthermore, we ran a randomized controlled experiment to explore the impact of highlighting important words detected by DL-based graders on human grading. The results showed that: (i) DL-based graders, to a certain degree, displayed alignment with human graders no matter whether DL-based graders and human graders agreed on the quality of an answer; and (ii) it is possible to facilitate human grading by highlighting those DL-detected important words, though further investigations are necessary to understand how human graders exploit such highlighted words.

show abstract