Automated Cross-prompt Scoring of Essay Traits

Abstract

The majority of current research in Automated Essay Scoring (AES) focuses on prompt-specific scoring of either the overall quality of an essay or the quality with regards to certain traits. In real-world applications obtaining labelled data for a target essay prompt is often expensive or unfeasible, requiring the AES system to be able to perform well when predicting scores for essays from unseen prompts. As a result, some recent research has been dedicated to cross-prompt AES. However, this line of research has thus far only been concerned with holistic, overall scoring, with no exploration into the scoring of different traits. As users of AES systems often require feedback with regards to different aspects of their writing, trait scoring is a necessary component of an effective AES system. Therefore, to address this need, we introduce a new task named Automated Cross-prompt Scoring of Essay Traits, which requires the model to be trained solely on non-target-prompt essays and to predict the holistic, overall score as well as scores for a number of specific traits for target-prompt essays. This task challenges the model's ability to generalize in order to score essays from a novel domain as well as its ability to represent the quality of essays from multiple different aspects. In addition, we introduce a new, innovative approach which builds on top of a state-of-the-art method for cross-prompt AES. Our method utilizes a trait-attention mechanism and a multi-task architecture that leverages the relationships between each trait to simultaneously predict the overall score and the score of each individual trait. We conduct extensive experiments on the widely used ASAP and ASAP++ datasets and demonstrate that our approach is able to outperform leading prompt-specific trait scoring and cross-prompt AES methods.

Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay grading

“…Researchers have proposed innovative methodologies for generalizing essay scores across various prompts and domains [29][30][31]. For example, [32] presented a two-stage deep neural network that generated pseudo-data for both prompt-dependent and prompt-specific conditions, thereby enhancing the model adaptability across different prompts.…”

Integration of Prediction Scores From Various Automated Essay Scoring Models Using Item Response Theory

“…The feature-engineering and DNN-based automatic feature extraction approaches can be viewed as complementary rather than competing [6] because they have different advantages and drawbacks. Thus, some hybrid models that integrate the two approaches have recently been proposed [10], [25], [42], [44], [46].…”

“…Then, the concatenated vector is mapped to a score value through a regression layer, such as a linear layer with sigmoid activation. The DNN-AES models used in the hybrid models include variants of the CNN-RNNbased model [10], [25] and the BERT-based model [42]. As an example, Appendix C introduces details on the BERT-based hybrid AES model.…”

“…These DNN-AES models predict a score from the sequence of words in the essay, meaning that no manually designed features are required. However, some recent studies have proposed hybrid models that incorporate manually designed features into DNN-AES models [10], [25], [42], [44], [46] and have reported that such a hybrid approach is effective for improving scoring accuracy.…”

Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring

“…Acknowledging this, recent works have suggested cross-prompt models (Jin et al, 2018;Li et al, 2020;Ridley et al, 2020) that are tested using essays of unseen prompt, like zero-shot learning, and trait-scoring models (Mathias and Bhattacharyya, 2020;Hussein et al, 2020;Kumar et al, 2021;He et al, 2022) that output multiple trait scores. Handling both settings (Figure 1) is a direction for practical AES and yet has rarely been studied (Ridley et al, 2021).…”