Automated Versus Human Essay Scoring: A Comparative Study

Abstract

This study investigated the effects of automated essay scoring (AES) system on writing improvement of Iranian L2 learners. About 60 Iranian intermediate EFL learners were selected on a Standard English proficiency test (Allen 2004). Afterwards, they were randomly assigned to two groups of 30, experimental and control group. Participants in experimental group received the AES scoring, and control group, received the human scoring. Statistical analyses of the results reveal that 1) AES tool results in significant improvement of L2 learners writing achievement, 2) Results from questionnaire show that Students ware favor about using AES tool, 3) The results from the current study support the conclusion that the AES tool does not seem to correlate well with human raters in scoring essays. Hence, the findings of this study indicate that using AES tools can help teachers ease their big teaching students to improve their writing and it can be used as an educational tool on classrooms

A Study on the Effectiveness of Automated Essay Marking in the Context of a Blended Learning Course Design

“…Previous studies have shown that high correlation can be achieved between manual scoring system and AES system (Kukich, 2000;Attali & Burstein, 2006;Ben-Simon & Bennett, 2007;Toranj & Ansari, 2012).…”

“…From then on, some commercial and also several non-profit organizations took up exploring different types of essay scoring systems for English language. AES systems at that time were adopted by testing companies, universities, and public schools (Toranj & Ansari, 2012). The most widely known AES systems include Project Essay Grader (Page, 1966(Page, , 1968(Page, , 2003 became commercially competitive around this time as it was able to combine the teaching of English writing and the development of large-scale tests of writing.…”

“…Studies on AES systems have demonstrated that computers can function as more effective cognitive tools (Joundy et al, 2019;Attali, 2004;Toranj & Ansari, 2012). Researchers have found that the AES system could be useful as it was able to give scores and feedback to students rapidly (Page, 2003).…”

Penggunaan Penskor Jawaban Esai Otomatis dalam Pengukuran Pengetahuan Guru

“…Model penskor esai otomatis umumnya divalidasi dengan cara membandingkan kesamaan hasil penskorannya dengan hasil penskoran manusia untuk data yang sama (Attali & Burstein, 2006;Chung & Baker, 2003;Graham, dkk,, 2012;Williamson, dkk., 2010). Rater atau penskor manusia dianggap sebagai standar untuk penskoran yang akan dilakukan lebih luas oleh penskor otomatis, artinya hasil perbandingan kesamaan menjadi penentu kualitas penskor otomatis (Powers, dkk., 2015;Shermi,s dkk., 2010;Toranj & Ansari, 2012;Wahlen, dkk., 2020;Williamson, dkk., 2010;Zhang, 2013). Korelasi antara program penskoran esai otomatis dan penilai manusia akan bervariasi tergantung pada kalibrasi yang dilakukan, sampel yang digunakan untuk menghitung korelasi, atau validasi silang yang digunakan (Keith, 2003).…”

Semantic Document Networks to Support Concept Retrieval

“…process is important in information retrieval, meaningful retrieval of stored information from a data repository with digital artefacts (Micarelli, Sciarrone, & Marinilli, 2007); automated essay grading, to compare essays using their 'feature space' of relevant concepts and their interrelationships and comparing these to model answers (Toranj & Ansari, 2012;Dikli, 2006); detection of plagiarism, identifying reproduced ideas and structuring of concepts as opposed to identifying perfectly copied segments of text (Osman, Salim, Binwahlan, Alteeb, & Abuobieda, 2012); and ontology construction, where an ontology is formed from text relations and both generalization and specialization of concepts (Buitelaar, Cimiano, & Magnini, 2005), using the complex interplay between these relations to construct an ontology schema.…”