Development and Evaluation of a Custom GPT for the Assessment of Students’ Designs in a Typography Course

Abstract

The recent advancements in the fields of AI technology, generative AI, and Large Language Models (LLMs) have increased the potential of the deployment of such tools in educational environments, especially in contexts where student assessment fairness, quality, and automation are a priority. This study introduces an AI-enhanced evaluation tool that utilizes OpenAI’s GPT-4 and the recently released custom GPT feature to evaluate the typography designs of 25 students enrolled in the Visual Media diploma offered by King Abdulaziz University. A mixed methods approach is adopted to evaluate the performance of this tool against the rubric-based evaluations offered by two human evaluators, considering both grading and text feedback. The results indicate that there are statistically significant differences between the AI tool’s grading and feedback when compared to that of Evaluator 2; however, none is reported with Evaluator 1. The study presents a qualitative interpretation of the comprehensive feedback by the evaluator and reflects in further research in this area.

Rubric Development and Validation for Assessing Tasks' Solving via AI Chatbots

“…Unlike previous research by García-Orosa, Canavilhas and Vázquez-Herrero (2023) and Yang ( 2022), which emphasized the potential and challenges of integrating AI in education without a clear framework for evaluation, this study provides a concrete rubric for assessing the effectiveness of AI chatbots in educational settings. The rubric's focus on "Accuracy," "Relevance," and "Efficiency" parallels the attributes identified by Almasre (2024) and Cope, Kalantzis, and Searsmith (2021) as essential for educational tools. However, our findings extend beyond these attributes by validating a comprehensive set of criteria through empirical methods, addressing a gap in the literature regarding the systematic assessment of AI chatbots.…”

“…These results go beyond previous reports. For instance, the validation of the rubric, underscored by the unanimous acceptance of criteria such as "Accuracy," "Relevance," and "Efficiency," echoes the critical attributes highlighted in the literature for evaluating educational tools (Almasre, 2024;Cope, Kalantzis, and Searsmith, 2021). These attributes are important to ensure that AI chatbots effectively aid pedagogy, fostering both knowledge acquisition and the development of critical thinking skills among students.…”