Chatbot-based serious games: A useful tool for training medical students? A randomized controlled trial

Abstract

Objectives Chatbots, conversational agents that walk medical students (MS) though a clinical case, are serious games that seem to be appreciated by MS. Their impact on MS’s performance in exams however was not yet evaluated. Chatprogress is a chatbot-based game developed at Paris Descartes University. It contains 8 pulmonology cases with step-by-step answers delivered with pedagogical comments. The CHATPROGRESS study aimed to evaluate the impact of Chatprogress on students’ success rate in their end-term exams. Methods We conducted a post-test randomized controlled trial held on all fourth-year MS at Paris Descartes University. All MS were asked to follow the University’s regular lectures, and half of them were randomly given access to Chatprogress. At the end of the term, medical students were evaluated on pulmonology, cardiology and critical care medicine. Main outcomes measures The primary aim was to evaluate an increase in scores in the pulmonology sub-test for students who had access to Chatprogress, compared to those who didn’t. Secondary aims were to evaluate an increase in scores in the overall test (Pulmonology, Cardiology and Critical care medicine test (PCC)) and to evaluate the correlation between access to Chatprogress and overall test score. Finally, students’ satisfaction was assessed using a survey. Results From 10/2018 to 06/2019, 171 students had access to Chatprogress (the Gamers) and among them, 104 ended up using it (the Users). Gamers and Users were compared to 255 Controls with no access to Chatprogress. Differences in scores on the pulmonology sub-test over the academic year were significantly higher among Gamers and Users vs Controls (mean score: 12.7/20 vs 12.0/20, p = 0.0104 and mean score: 12.7/20 vs 12.0/20, p = 0.0365 respectively). This significant difference was present as well in the overall PCC test scores: (mean score: 12.5/20 vs 12.1/20, p = 0.0285 and 12.6/20 vs 12.1/20, p = 0.0355 respectively). Although no significant correlation was found between the pulmonology sub-test’s scores and MS’s assiduity parameters (number of finished games among the 8 proposed to Users and number of times a User finished a game), there was a trend to a better correlation when users were evaluated on a subject covered by Chatprogress. MS were also found to be fans of this teaching tool, asking for more pedagogical comments even when they got the questions right. Conclusion This randomised controlled trial is the first to demonstrate a significant improvement in students’ results (in both the pulmonology subtest and the overall PCC exam) when they had access to Chatbots, and even more so when they actually used it.

Implementation of a Virtual Patient Chatbot for Physiotherapy Students Training

“…In another study, students practicing their skills using chatbots did significantly better in subtests and overall test scores [10]. There was a better correlation in the test scores when the test items were similar to those covered by the chatbot.…”

“…In terms of learning, Chatprogress, a serious game chatbot, helped students to walk through clinical cases [10]. When students chose a wrong answer, the game ended, and they received an explanation and a reference from the textbook.…”

Artificial Intelligence Revolutionizing the Field of Medical Education

“…These can serve as preclinical interactive tutors and ward assistants in clinical settings to help students understand complex clinical scenarios and assist in developing decision-making skills [7]. The primary benefits of using chatbots include integrating content, quick access, allowing multiple users and immediate assistance [9,10]. Despite these, presently available chatbots can potentially create false references, a phenomenon described as "hallucination" [7].…”

Application of the Chatbot in University Education: A Systematic Review on the Acceptance and Impact on Learning

“…Chatbot interface 81.4% of students were satisfied with the interaction of the interface TO (Neo, 2022) 70% of students were attracted to the chatbot interface, so they used it again for their medical learning TO (Al Kahf et al, 2023) Nursing students were satisfied with the chatbot interface, obtaining a mean of 4.19 and a standard deviation of 0.72. NTO (Chang et al, 2022) Undergraduate students were satisfied with the chatbot interface, which helped them in their learning in finance.…”

“…(Al Kahf et al, 2023) The contribution of the chatbot in student learning resulted in an improvement in academic performance, with a mean variance of 0.78. (Malik et al, 2021) The chatbot's contribution to learning a second language, such as English, allowed undergraduate students to improve their academic performance.…”