Automated language essay scoring systems: A literature review

Abstract

Background. Writing composition is a significant factor for measuring test-takers’ ability in any language exam. However, the assessment (scoring) of these writing compositions or essays is a very challenging process in terms of reliability and time. The need for objective and quick scores has raised the need for a computer system that can automatically grade essay questions targeting specific prompt. Automated Essay Scoring (AES) systems are used to overcome the challenges of scoring writing tasks by using Natural Language Processing and Machine Learning techniques. The purpose of this paper is to review the literature for the AES systems used for grading the essay questions. Methodology. We have reviewed the existing literature using Google Scholar, EBSCO and ERIC to search the terms “AES”, “Automated Essay Scoring”, “Automated Essay Grading”, or “Automatic Essay”, and two categories have been identified: handcrafted features and automatic featuring AES systems. The systems of the first category are closely bonded to the quality of the designed features. On the other hand, the systems of the other category are based on the automatic learning of the features and relations between an essay and its score without any handcrafted features. We reviewed the systems of the two categories in terms of system primary focus, technique(s) used in the system, training data (y/n), instructional application (feedback system), and the correlation between e-scores and human scores. The paper is composed of three main sections. Firstly, we present a structured literature review of the available Handcrafted Features AES systems. Secondly, we present a structured literature review of the available Automatic Featuring AES systems. Finally, we draw a set of discussions and conclusions. Results. AES models have been found to utilize a broad range of manually-tuned shallow and deep linguistic features. AES systems have many strengths in reducing labour-intensive marking activities, ensuring a consistent application of marking criteria, and facilitating equity in scoring. Although many techniques have been implemented to improve the AES systems, three primary challenges have been concluded: they lack the sense of the rater as a person, they can be tricked into assigning a lower or higher score to an essay than it deserved or not, and they cannot assess the creativity of the ideas and propositions and evaluating their practicality. Many techniques have been used to address the first two challenges only.

The development of a collection of design fictions about Artificial Intelligence and robots in Higher Education

“…o One emerging area of adaptivity is using sensors to detect the emotional and physical state of the learner, recognising the embodied and affective aspects of learning (Luckin, et al, 2016); a further link is being made to how virtual and augmented reality can be used to make the experience more engaging and authentic (Holmes et al, 2019). • Automatic Writing Evaluation (AWE) which are tools to assess and offer feedback on writing style (rather than content) such as learnandwrite, Grammarly and Turnitin's Revision Assistant (Strobl, et al 2019;Hussein et al, 2019;Hockly, 2018). • Conversational agents (also known as Chatbots or virtual assistants) which are AI tools designed to converse with humans (Winkler and Sӧllner, 2018).…”

“…As the quotation from the fiction illustrates, echoing Bayne (2015), the conversation in Fiction 2 is not necessarily smooth; misunderstandings and conflicts occur. The fiction brings into view the less compliant vision of the student who might wish to game the system, a potential problem with AI which is apparent in the literature of AWE (Hussein et al 2019). This fiction encapsulates an important alternative potential imaginary of AI, as a simple, low-tech intervention.…”

An exploration of automated narrative analysis via machine learning

“…Such systems operate on predictive models, which analyze text and output standardized scores at varying levels of predictive accuracy, as measured by their correlation with human-produced scores [1]. Many other AES systems have been developed s ince the introduction of PEG to increase both the accuracy of predictive models, as well as to offer additional features, such as plagiarism checks and critical feedback [1, 7].…”

“…There are currently two main domains of AES systems: those that utilize hand-engineered essay features and those that utilize raw-text approaches [1, 7]. Hand-engineered features have long been the standard approach as they use knowledge provided by a domain expert to create numeric representations of the text, allowing for simple statistical modeling techniques to identify relevant patterns for scoring.…”

“…Using these vectors, machine learning (ML) methods can be applied, either to the sequence of text in order or to the “bag-of-words”, to construct a model for scoring the text. [7]…”