A Systematic Review of the Effects of Automatic Scoring and Automatic Feedback in Educational Settings

Abstract

Automatic scoring and feedback tools have become critical components of online learning proliferation. These tools range from multiple-choice questions to grading essays using machine learning (ML). Learning environments such as massive open online courses (MOOCs) would not be possible without them. The usage of this mechanism has brought many exciting areas of study, from the design of questions to the ML grading tools' precision and accuracy. This paper analyzes the findings of 125 studies published in journals and proceedings between 2016 and 2020 on the usages of automatic scoring and feedback as a learning tool. This analysis gives an overview of the trends, challenges, and open questions in this research area. The results indicate that automatic scoring and feedback have many advantages. The most important benefits include enabling scaling the number of students without adding a proportional number of instructors, improving the student experience by reducing the time between submission grading and feedback, and removing bias in scoring.On the other hand, these technologies have some drawbacks. The main problem is creating a disincentive to develop innovative answers that do not match the expected one or have not been considered when preparing the problem. Another drawback is potentially training the student to answer the question instead of learning the concepts. With this, given the exitance of a correct answer, such an answer could be leaked to the internet, making it easier for students to avoid solving the problem. Overall, each of these drawbacks presents an opportunity to look at ways to improve technologies to use these tools to provide a better learning experience to students.

Classtime.Com as an Ai-Based Testing Platform: Analysing ESP Students’ Performances and Feedback

“…The results of this study also imply that AI-based online testing platforms can be useful in language education by offering students flexibility, convenience, ease of use, engagement, immediate feedback, accessibility, efficiency, enjoyment, clarity, and special features (Hahn et al, 2021;Isnaini et al, 2021;Jin, 2022;Saienko & Chugai, 2020). However, the student feedback also emphasises the necessity of fixing technological issues, offering clear instructions and explanations, making sure the system is user-friendly, and enhancing the exam's intrigue and excitement.…”

“…In recent years, the utilization of Artificial Intelligence (AI) technology in the field of education, specifically in the realm of online assessment, has witnessed a notable increase (Chen et al, 2020;González-Calatayud et al, 2021;Huang et al, 2022). Scholars have acknowledged the potential benefits associated with the integration of AI in educational settings, with particular attention given to the advantages of automated scoring and immediate feedback, especially in the context of the COVID-19 pandemic (Hahn et al, 2021). However, despite these advancements, traditional paper-based assessment methods still prevail, presenting challenges for students (Nardi & Ranieri, 2019).…”

A comparison of various machine learning algorithms and execution of flask deployment on essay grading

“…It also presented a similarity checking algorithm for equations. The advantages and disadvantages of automatic scoring and feedback is systematically reviewed in [24].…”

The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments

“…Mixed-format assessments provide the benefits associated with both item formats, but they are also prone to unique psychometric challenges that should be addressed to minimize threats to the validity, reliability, and fairness of the assessment. One important psychometric consideration related to mixed-format assessments is that scoring of CR items mainly relies on human judgment, even when automated scoring processes are used (Hahn et al, 2021; Hussein et al, 2019). Even with rater training, the rating process for performance assessments is relatively subjective because it is based on raters’ own understanding of assessment context, interpretations of rubrics, expectations of performance, and professional experiences (Wu, 2017).…”