Incorporating AI and learning analytics to build trustworthy peer assessment systems

Abstract

Peer assessment has been recognised as a sustainable and scalable assessment method that promotes higher‐order learning and provides students with fast and detailed feedback on their work. Despite these benefits, some common concerns and criticisms are associated with the use of peer assessments (eg, scarcity of high‐quality feedback from peer student‐assessors and lack of accuracy in assigning a grade to the assessee) that raise questions about their trustworthiness. Consequently, many instructors and educational institutions have been anxious about incorporating peer assessment into their teaching. This paper aims to contribute to the growing literature on how AI and learning analytics may be incorporated to address some of the common concerns associated with peer assessment systems, which in turn can increase their trustworthiness and adoption. In particular, we present and evaluate our AI‐assisted and analytical approaches that aim to (1) offer guidelines and assistance to student‐assessors during individual reviews to provide better feedback, (2) integrate probabilistic and text analysis inference models to improve the accuracy of the assigned grades, (3) develop feedback on reviews strategies that enable peer assessors to review the work of each other, and (4) employ a spot‐checking mechanism to assist instructors in optimally overseeing the peer assessment process. What is already known about this topic Engaging students in peer assessment has been demonstrated to have various benefits. However, there are some common concerns associated with employing peer assessment that raise questions about their trustworthiness as an assessment item. What this paper adds Methods and processes on how AI and learning analytics may be incorporated to address some of the common concerns associated with peer assessment systems, which in turn, can increase their trustworthiness and adoption. Implications for practice Presentation of a systematic approach for development, deployment and evaluation of AI and analytics approaches in peer assessment systems.

Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models

“…Although the considerable amount of false positives and a few borderline false negatives in Fig. 5 demonstrate an evident necessity for instructors' support during the moderation process, which is aligned with the findings of previous studies [10], [97], [98], the use of advanced consensus approaches can significantly reduce the instructors' load in the quality assessment of SGC at scale.…”

Integration of artificial intelligence performance prediction and learning analytics to improve student learning in online engineering course

“…Particularly, the integration of AI and LA has potential to provide both quantitative performance generated by AI model and qualitative feedback from the instructor’s or researcher’s perspectives, which can further improve student learning process and performance. For example, Darvishi et al ( 2022 ) incorporated AI and LA to build a trustworthy peer assessment system and this research showed that this AI-assisted and analytical approaches integrated AI model to improve the accuracy of the assigned task for the instructors. Students, supported with the integrated AI and LA techniques, successfully wrote lengthier and more helpful feedback comments to peers, which improved their learning effects and final performances.…”

“…Particularly, AI can automatically capture and analyze the learning process and the learner's psychological states, and LA can provide relevant feedback and suggestions from the educators or practitioners concerning the cognitive process, social interaction, and affectional or metacognitive states (Starcic, 2019). More importantly, relevant literature indicates that the integrated AI and LA techniques have potential to eventually improve students' learning performances (e.g., Chango et al, 2021;Darvishi et al, 2022).…”

“…Furthermore, although there is a research trend to carrying out on the educational application of AI prediction models in practice, there is a lack of applying AI prediction model that integrates in-time feedback and offers appropriate feedback to the instructors and students in order to improve students’ learning quality. The recent research trend of the integration of AI and learning analytics (LA) has potential to address this issue (Darvishi et al, 2022 ; de Carvalho & Zárate, 2020 ; Starcic, 2019 ).…”

Technological frameworks on ethical and trustworthy learning analytics

“…Moreover, in certain cases, utility tends to increase when LA models are built without protected attributes, such as gender or socio‐economic status. Finally, Darvishi et al (2022) present a systematic approach of using LA and AI‐based methods in peer assessment processes, aiming at increasing trustworthiness, effectiveness, and efficiency, which in turn might enhance adoption of such assessment processes. Darvishi and colleagues show results on a large dataset using a concrete adaptive system with peer assessment functionalities over several courses.…”