Performance of the Large Language Model ChatGPT on the National Nurse Examinations in Japan: Evaluation Study

Abstract

Background ChatGPT, a large language model, has shown good performance on physician certification examinations and medical consultations. However, its performance has not been examined in languages other than English or on nursing examinations. Objective We aimed to evaluate the performance of ChatGPT on the Japanese National Nurse Examinations. Methods We evaluated the percentages of correct answers provided by ChatGPT (GPT-3.5) for all questions on the Japanese National Nurse Examinations from 2019 to 2023, excluding inappropriate questions and those containing images. Inappropriate questions were pointed out by a third-party organization and announced by the government to be excluded from scoring. Specifically, these include “questions with inappropriate question difficulty” and “questions with errors in the questions or choices.” These examinations consist of 240 questions each year, divided into basic knowledge questions that test the basic issues of particular importance to nurses and general questions that test a wide range of specialized knowledge. Furthermore, the questions had 2 types of formats: simple-choice and situation-setup questions. Simple-choice questions are primarily knowledge-based and multiple-choice, whereas situation-setup questions entail the candidate reading a patient’s and family situation’s description, and selecting the nurse's action or patient's response. Hence, the questions were standardized using 2 types of prompts before requesting answers from ChatGPT. Chi-square tests were conducted to compare the percentage of correct answers for each year's examination format and specialty area related to the question. In addition, a Cochran-Armitage trend test was performed with the percentage of correct answers from 2019 to 2023. Results The 5-year average percentage of correct answers for ChatGPT was 75.1% (SD 3%) for basic knowledge questions and 64.5% (SD 5%) for general questions. The highest percentage of correct answers on the 2019 examination was 80% for basic knowledge questions and 71.2% for general questions. ChatGPT met the passing criteria for the 2019 Japanese National Nurse Examination and was close to passing the 2020-2023 examinations, with only a few more correct answers required to pass. ChatGPT had a lower percentage of correct answers in some areas, such as pharmacology, social welfare, related law and regulations, endocrinology/metabolism, and dermatology, and a higher percentage of correct answers in the areas of nutrition, pathology, hematology, ophthalmology, otolaryngology, dentistry and dental surgery, and nursing integration and practice. Conclusions ChatGPT only passed the 2019 Japanese National Nursing Examination during the most recent 5 years. Although it did not pass the examinations from other years, it performed very close to the passing level, even in those containing questions related to psychology, communication, and nursing.

Superior Performance of Artificial Intelligence Models in English Compared to Arabic in Infectious Disease Queries

“…Several studies assessed AI model performance in non-English languages with variable results despite the overall trend of below bar performance in non-English languages. For example, Taira et al tested ChatGPT performance in the Japanese National Nursing Examination in Japanese language in ve consecutive years [43]. Despite approaching the passing threshold in four years and passing the 2019 exam, the results indicated the relative weakness of ChatGPT in Japanese [43].…”

“…For example, Taira et al tested ChatGPT performance in the Japanese National Nursing Examination in Japanese language in ve consecutive years [43]. Despite approaching the passing threshold in four years and passing the 2019 exam, the results indicated the relative weakness of ChatGPT in Japanese [43]. Nevertheless, attributing this result to language limitations alone is challenging, given the superior performance of ChatGPT-4 in Japanese language compared to medical residents in the Japanese General Medicine In-Training Examination, as reported by Watari et al [44].…”

Performance Comparison of ChatGPT-4 and Japanese Medical Residents in the General Medicine In-Training Examination: Comparison Study

“…Consistently, prior research on the Japanese national medical examinations found that the performance gap between AI and humans widened with increasing question difficulty [ 12 ]. Indeed, AI models such as GPT-4 have achieved the proficiency level required to pass even highly challenging certification examinations that often pose challenges for many humans [ 2 - 5 , 11 , 12 ]. Because common clinical scenarios often follow a distinct framework or pattern, AI’s rule-based responses have the potential to surpass human performance [ 22 , 23 ].…”

“…This assessment is especially relevant because Japanese is considered among English natives as one of the most challenging languages to master [ 10 ]. Interestingly, it has been suggested that GPT-3.5, the precursor to GPT-4, has achieved passing grades on the Japanese Nursing Licensing examination [ 11 ]. In the latest Japanese national medical licensing examination in February 2023, GPT-4 attained passing levels while GPT-3.5 showed that it is not far behind the passing criteria [ 12 ].…”

Embracing the future: Integrating ChatGPT into China’s nursing education system

“…Research has shown that ChatGPT can assist nurses in medical record documentation [ 37 ], enhance patient education resources [ 38 ], and successfully boost patient communication efficiency [ 37 ]. One study discovered that ChatGPT successfully passed the Japanese registered nursing licensure test [ 39 ]. While experienced nursing educators produce National Council Licensure Examination for Registered Nurses (NCLEX-RN) questions for daily test practice, the actual NCLEX-RN exam uses computer-generated questions based on the college’s current response scenario.…”