ReaderBench Learns Dutch: Building a Comprehensive Automated Essay Scoring System for Dutch Language

Dascălu, Mihai; Westera, Wim; Ruşeţi, Ştefan; Trăușan-Matu, Ștefan; Kurvers, Hub

doi:10.1007/978-3-319-61425-0_5

Cited by 15 publications

(10 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we investigate how automated essay scoring methods can help to reduce the teachers´ work load. In a preliminary study the automated essay scoring method for the reports in the VIBOA game demonstrated a validity level of 75.1% in correctly predicting a pass or a failure (Dascalu, Westera, Ruseti, Trausan-Matu, & Kurvers, 2017). Although this rate is substantial, it is considered insufficient for practical application.…”

Section: Introductionmentioning

confidence: 99%

Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training

Westera

Dascălu

Kurvers

et al. 2018

Computers & Education

Self Cite

View full text Add to dashboard Cite

This paper presents a methodology for applying automated essay scoring in educational settings. The methodology was tested and validated on a dataset of 173 reports (in Dutch language) that students have created in an applied game on environmental policy. Natural Language Processing technologies from the ReaderBench framework were used to generate an extensive set of textual complexity indices for each of the reports. Afterwards, different machine learning algorithms were used to predict the scores. By combining binary classification (pass or fail) and a probabilistic model for precision, a trade-off can be made between validity of automated score prediction (precision) and the reduction of teacher workload required for manual assessment. It was found from the sample that substantial workload reduction can be achieved, while preserving high precision: allowing for a precision of 95% or higher would already reduce the teacher's workload to 74%; lowering precision to 80% produces a workload reduction of 50%.

show abstract

Section: Introductionmentioning

confidence: 99%

Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training

Westera

Dascălu

Kurvers

et al. 2018

Computers & Education

Self Cite

View full text Add to dashboard Cite

show abstract

“…Many AES methods have been developed over the past decades, and can generally be categorized as feature-engineering and neural-network approaches (Hussein et al, 2019;Ke and Ng, 2019). The featureengineering approach predicts scores using handcrafted features such as essay length or spelling errors (e.g., (Amorim et al, 2018;Dascalu et al, 2017;Mark D. Shermis, 2016;Nguyen and Litman, 2018)). The advantages of this approach include interpretability and explainability.…”

Section: Introductionmentioning

confidence: 99%

Neural Automated Essay Scoring Incorporating Handcrafted Features

Uto¹,

Xie²,

Ueno³

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Automated essay scoring (AES) is the task of automatically assigning scores to essays as an alternative to grading by human raters. Conventional AES typically relies on handcrafted features, whereas recent studies have proposed AES models based on deep neural networks (DNNs) to obviate the need for feature engineering. Furthermore, hybrid methods that integrate handcrafted features in a DNN-AES model have been recently developed and have achieved state-of-the-art accuracy. One of the most popular hybrid methods is formulated as a DNN-AES model with an additional recurrent neural network (RNN) that processes a sequence of handcrafted sentencelevel features. However, this method has the following problems: 1) It cannot incorporate effective essay-level features developed in previous AES research. 2) It greatly increases the numbers of model parameters and tuning parameters, increasing the difficulty of model training. 3) It has an additional RNN to process sentence-level features, enabling extension to various DNN-AES models complex. To resolve these problems, we propose a new hybrid method that integrates handcrafted essay-level features into a DNN-AES model. Specifically, our method concatenates handcrafted essay-level features to a distributed essay representation vector, which is obtained from an intermediate layer of a DNN-AES model. Our method is a simple DNN-AES extension, but significantly improves scoring accuracy.

show abstract

“…Several existing text analysis tools can calculate a huge number of textual features against input texts. ReaderBench [22] is an open source multilingual framework that makes use of natural language processing techniques to provide text analysis tools. The framework is multilingual [23] -text analysis tools are available in Dutch, French, Romanian and English.…”

Section: Introductionmentioning

confidence: 99%

Identifying Critical Features for Formative Essay Feedback with Artificial Neural Networks and Backward Elimination

Abbas

Rosmalen

Kalz

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

For predicting and improving the quality of essays, text analytic metrics (surface, syntactic, morphological and semantic features) can be used to provide formative feedback to the students. In this study, the intent was to find a small number of features that exhibit a fair proxy of the scores given by the human raters. Using an existing corpus and a text analysis tool for the Dutch language, a large number of features were extracted. Artificial neural networks, Levenberg Marquardt algorithm and backward elimination were used to reduce the number of extracted features automatically. Irrelevant features were eliminated based on the inter-rater agreement between predicted and human scores calculated using Cohen's Kappa (κ). By using our algorithm, the number of features in this study was reduced from 457 to 23. The selected features were grouped into six different categories. Of these categories, we believe that the features present in the groups "Word Difficulty" and "Lexical Diversity" are most useful for providing automated formative feedback to the students. The approach presented in this research paper is the first step towards our ultimate goal of providing meaningful formative feedback to the students for enhancing their writing skills and capabilities.

show abstract

ReaderBench Learns Dutch: Building a Comprehensive Automated Essay Scoring System for Dutch Language

Cited by 15 publications

References 26 publications

Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training

Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training

Neural Automated Essay Scoring Incorporating Handcrafted Features

Identifying Critical Features for Formative Essay Feedback with Artificial Neural Networks and Backward Elimination

Contact Info

Product

Resources

About