Sailing the Seven Seas: A Multinational Comparison of ChatGPT’s Performance on Medical Licensing Examinations

Abstract

Purpose The use of AI-powered technology, particularly OpenAI’s ChatGPT, holds significant potential to reshape healthcare and medical education. Despite existing studies on the performance of ChatGPT in medical licensing examinations across different nations, a comprehensive, multinational analysis using rigorous methodology is currently lacking. Our study sought to address this gap by evaluating the performance of ChatGPT on six different national medical licensing exams and investigating the relationship between test question length and ChatGPT’s accuracy. Methods We manually inputted a total of 1,800 test questions (300 each from US, Italian, French, Spanish, UK, and Indian medical licensing examination) into ChatGPT, and recorded the accuracy of its responses. Results We found significant variance in ChatGPT’s test accuracy across different countries, with the highest accuracy seen in the Italian examination (73% correct answers) and the lowest in the French examination (22% correct answers). Interestingly, question length correlated with ChatGPT’s performance in the Italian and French state examinations only. In addition, the study revealed that questions requiring multiple correct answers, as seen in the French examination, posed a greater challenge to ChatGPT. Conclusion Our findings underscore the need for future research to further delineate ChatGPT’s strengths and limitations in medical test-taking across additional countries and to develop guidelines to prevent AI-assisted cheating in medical examinations.

Below average ChatGPT performance in medical microbiology exam compared to university students

“…Therefore, more studies are needed to reach reliable conclusions about the performance of AI-based models in comparison to the performance of students in higher education. Additionally, it is important to consider the performance of the AI-based models taking into account the different language and cultural factors similar to the approach taken by Alfertshofer et al that tested ChatGPT performance across the US, Italian, French, Spanish, UK, and Indian medical licensing exams (Alfertshofer et al, 2023).…”

“…An emerging research field that gained a huge momentum recently involved the exploration of AI-based conversational models' ability to answer MCQs (Alfertshofer et al, 2023;Chang et al, 2023;Fuchs et al, 2023;Giannos and Delardas, 2023;Oztermeli and Oztermeli, 2023). The motivation of this research endeavor can be related to several aspects as follows.…”

A multinational study on the factors influencing university students’ attitudes and usage of ChatGPT

“…Meyer et al emphasized the importance of students' engagement as prompt creators and fact checkers in an educational framework, rather than simply relying on AI-produced material 63 . Multiple recent studies highlighted the need to revise the current assessment methods in higher education in light of the high performance of LLMs in various exams [66][67][68][69] . www.nature.com/scientificreports/ Based on the prospects of ChatGPT in higher education, a previous study explored the validity of a survey instrument to assess the factors influencing the adoption of this novel tool among university students in health schools in Jordan 51 .…”

Artificial Intelligence as a Triage Tool during the Perioperative Period: Pilot Study of Accuracy and Accessibility for Clinical Application

“…Such findings and proof of its working knowledge base are corroborated by its successful performance on examinations in plastic surgery and other specialties. 15,18,19,29 In addition to the integrity and accuracy of medical advice, accessibility is an important concern. Baseline health literacy levels have been demonstrated to be inadequate amongst plastic surgery patients.…”

“…Such findings and proof of its working knowledge base are corroborated by its successful performance on examinations in plastic surgery and other specialties. 15,18,19,29…”