Exploring the proficiency of ChatGPT-4: An evaluation of its performance in the Taiwan advanced medical licensing examination

Abstract

Background Taiwan is well-known for its quality healthcare system. The country's medical licensing exams offer a way to evaluate ChatGPT's medical proficiency. Methods We analyzed exam data from February 2022, July 2022, February 2023, and July 2033. Each exam included four papers with 80 single-choice questions, grouped as descriptive or picture-based. We used ChatGPT-4 for evaluation. Incorrect answers prompted a “chain of thought” approach. Accuracy rates were calculated as percentages. Results ChatGPT-4's accuracy in medical exams ranged from 63.75% to 93.75% (February 2022–July 2023). The highest accuracy (93.75%) was in February 2022's Medicine Exam (3). Subjects with the highest misanswered rates were ophthalmology (28.95%), breast surgery (27.27%), plastic surgery (26.67%), orthopedics (25.00%), and general surgery (24.59%). While using “chain of thought,” the “Accuracy of (CoT) prompting” ranged from 0.00% to 88.89%, and the final overall accuracy rate ranged from 90% to 98%. Conclusion ChatGPT-4 succeeded in Taiwan's medical licensing exams. With the “chain of thought” prompt, it improved accuracy to over 90%.

Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis

“…Researchers have not analyzed or elaborated on the impact of these task understanding prompts on ChatGPT's performance. However, three studies used optimized prompts [19,26,35]. A Korean study used four kinds of optimized prompts including: annotating Chinese terms in TKM, translating the instruction and question into English, providing exam-optimized instructions, and utilizing selfconsistency in the prompt.…”

“…Pretend to be a junior doctor with expertise in clinical practice and exam solving and retry" or "Could you double-check the answer?". ChatGPT could correctly answer up to 88.9% and 84% of these questions, respectively [19,35]. For task understanding prompts, we conducted a subgroup analysis and meta-regression to examine whether they affected ChatGPT's performance.…”

“…Although GPT-4 performed better overall than GPT-3.5 [33,36,41,44,47], it did not pass the Japanese medical licensing exam [49]. Additionally, ChatGPT performance varies significantly across medical specialties within these examinations [23,25,26,27,33,34,32,34,35]. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT's performance on different medical licensing exams.…”

Performance of GPT-4 and mainstream Chinese Large Language Models on the Chinese Postgraduate Examination dataset: Potential for AI-assisted Traditional Chinese Medicine

“…Multiple studies have systematically evaluated ChatGPT's performance on standardized tests across various languages. Notably, it has demonstrated excellent performance on assessments such as the United States Medical Licensing Examination (USMLE) [11][12][13] , the Japanese Medical Licensing Examination (JMLE) 14 , the Saudi Medical Licensing Examination (SMLE) 15 , the Polish medical specialization licensing exam (PES) 16 and Taiwan's medical licensing exams 17 . However, over the past ve years of the Chinese National Medical Licensing Examination (NMLE), ChatGPT scores have consistently fallen below the passing threshold.…”