Performance Comparison of ChatGPT-4 and Japanese Medical Residents in the General Medicine In-Training Examination: Comparison Study

Abstract

Background The reliability of GPT-4, a state-of-the-art expansive language model specializing in clinical reasoning and medical knowledge, remains largely unverified across non-English languages. Objective This study aims to compare fundamental clinical competencies between Japanese residents and GPT-4 by using the General Medicine In-Training Examination (GM-ITE). Methods We used the GPT-4 model provided by OpenAI and the GM-ITE examination questions for the years 2020, 2021, and 2022 to conduct a comparative analysis. This analysis focused on evaluating the performance of individuals who were concluding their second year of residency in comparison to that of GPT-4. Given the current abilities of GPT-4, our study included only single-choice exam questions, excluding those involving audio, video, or image data. The assessment included 4 categories: general theory (professionalism and medical interviewing), symptomatology and clinical reasoning, physical examinations and clinical procedures, and specific diseases. Additionally, we categorized the questions into 7 specialty fields and 3 levels of difficulty, which were determined based on residents’ correct response rates. Results Upon examination of 137 GM-ITE questions in Japanese, GPT-4 scores were significantly higher than the mean scores of residents (residents: 55.8%, GPT-4: 70.1%; P<.001). In terms of specific disciplines, GPT-4 scored 23.5 points higher in the “specific diseases,” 30.9 points higher in “obstetrics and gynecology,” and 26.1 points higher in “internal medicine.” In contrast, GPT-4 scores in “medical interviewing and professionalism,” “general practice,” and “psychiatry” were lower than those of the residents, although this discrepancy was not statistically significant. Upon analyzing scores based on question difficulty, GPT-4 scores were 17.2 points lower for easy problems (P=.007) but were 25.4 and 24.4 points higher for normal and difficult problems, respectively (P<.001). In year-on-year comparisons, GPT-4 scores were 21.7 and 21.5 points higher in the 2020 (P=.01) and 2022 (P=.003) examinations, respectively, but only 3.5 points higher in the 2021 examinations (no significant difference). Conclusions In the Japanese language, GPT-4 also outperformed the average medical residents in the GM-ITE test, originally designed for them. Specifically, GPT-4 demonstrated a tendency to score higher on difficult questions with low resident correct response rates and those demanding a more comprehensive understanding of diseases. However, GPT-4 scored comparatively lower on questions that residents could readily answer, such as those testing attitudes toward patients and professionalism, as well as those necessitating an understanding of context and communication. These findings highlight the strengths and limitations of artificial intelligence applications in medical education and practice.

Superior Performance of Artificial Intelligence Models in English Compared to Arabic in Infectious Disease Queries

“…Nevertheless, attributing this result to language limitations alone is challenging, given the superior performance of ChatGPT-4 in Japanese language compared to medical residents in the Japanese General Medicine In-Training Examination, as reported by Watari et al [44]. This study also exposed ChatGPT-4 limitations in test aspects requiring empathy, professionalism, and contextual understanding [44].…”

Evaluating ChatGPT performance in Arabic dialects: A comparative study showing defects in responding to Jordanian and Tunisian general health prompts

“…The results suggest that disparities in language performance, as evident in Arabic dialects, could potentially extend to other languages. This was shown in Japanese, French, and Polish languages among others [21,22,32]. Thus, collaborative efforts should be implemented to create diverse AI training datasets, which would help to ensure the generation of equitable and accurate health information across different linguistic and cultural contexts.…”

Performance of Artificial Intelligence: Does Artificial Intelligence Dream of Electric Sheep

“…Further, a previous study found that generative AI achieved a 79.9% correct response rate on the Japanese Medical Licensing Exam, notably outperforming the average examinee by 17% on hard questions (Takagi et 4 al., 2023). Furthermore, a prior study found that generative AI outperformed the Japanese medical residents on the General Medicine In-Training Examination, particularly in areas requiring detailed medical knowledge and difficult questions (Watari et al, 2023).…”

“…Building on a growing body of research on the performance of AI, our study addresses the first research question: Can generative AI evaluate the acceptance of generative AI as effectively as humans? Previous studies have shown that AI, including generative AI, performs impressively in various academic and professional exams, equalling or exceeding human performance across fields such as medicine, engineering, 16 and finance (Frieder et al, 2024;Gilson et al, 2023;Kung et al, 2023;Takagi et al, 2023;Terwiesch, 2023;Watari et al, 2023;Yang & Stivers, 2024). Moreover, prior studies find that generative AI can effectively perform tasks similar to humans, such as providing writing feedback, evaluating essays, and supporting language learning (Escalante et al, 2023;Guo & Wang, 2023;Mizumoto & Eguchi, 2023).…”