Impact of Generative AI on the Academic Performance and Test Anxiety of Pharmacy Students in OSCE: A Randomized Controlled Trial

Abstract

Background The rapid advancement of generative Artificial Intelligence (AI) in recent years has led to its increased application across various fields including education. One area where AI can significantly impact is clinical education, particularly in the preparation and execution of Objective Structured Clinical Examinations (OSCEs). This study aimed to evaluate the impact of AI-generated study material and feedback on the academic performance and level of anxiety of pharmacy students in formative OSCE. Method This was a 4-week (June-July 2024) randomized controlled study. Students of 6th semester PharmD program were randomized into either an intervention or control group. The intervention group received intervention which comprised a comprehensive training session on how to use AI tools (ChatGPT, Gemini and Perplexity) for generating study materials and practice OSCE stations with personalized feedback, in addition to usual OSCE instructions. The control group only received the usual OSCE instructions. In addition, all students completed the Test Anxiety Inventory (TAI) questionnaire before the OSCE. Result Eighty-eight (40 male, 48 female) out of 92 (96%) students attended the OSCE and completed the TAI questionnaire. Each group had 44 (50%) students. The mean OSCE mark was 13.26 (±5.05) out of 30. No significant difference was found between the intervention [12.98 (±5.15)] and control [13.54 (±5.00)] groups regarding mean OSCE marks (p=0.550). Similarly, no significant difference was found between the intervention and control groups regarding the emotionality subscale and worry subscale of the TAI (p=0.736; p=0.329) as well as the total TAI score (p=0.917). Conclusion While the use of AI tools did not improve the academic performance of students or reduce test-related anxiety, they did not negatively impact these outcomes either. Future research should investigate the long-term effects of AI-based interventions on educational outcomes.