Background ChatGPT, a large language model, has shown good performance on physician certification examinations and medical consultations. However, its performance has not been examined in languages other than English or on nursing examinations. Objective We aimed to evaluate the performance of ChatGPT on the Japanese National Nurse Examinations. Methods We evaluated the percentages of correct answers provided by ChatGPT (GPT-3.5) for all questions on the Japanese National Nurse Examinations from 2019 to 2023, excluding inappropriate questions and those containing images. Inappropriate questions were pointed out by a third-party organization and announced by the government to be excluded from scoring. Specifically, these include “questions with inappropriate question difficulty” and “questions with errors in the questions or choices.” These examinations consist of 240 questions each year, divided into basic knowledge questions that test the basic issues of particular importance to nurses and general questions that test a wide range of specialized knowledge. Furthermore, the questions had 2 types of formats: simple-choice and situation-setup questions. Simple-choice questions are primarily knowledge-based and multiple-choice, whereas situation-setup questions entail the candidate reading a patient’s and family situation’s description, and selecting the nurse's action or patient's response. Hence, the questions were standardized using 2 types of prompts before requesting answers from ChatGPT. Chi-square tests were conducted to compare the percentage of correct answers for each year's examination format and specialty area related to the question. In addition, a Cochran-Armitage trend test was performed with the percentage of correct answers from 2019 to 2023. Results The 5-year average percentage of correct answers for ChatGPT was 75.1% (SD 3%) for basic knowledge questions and 64.5% (SD 5%) for general questions. The highest percentage of correct answers on the 2019 examination was 80% for basic knowledge questions and 71.2% for general questions. ChatGPT met the passing criteria for the 2019 Japanese National Nurse Examination and was close to passing the 2020-2023 examinations, with only a few more correct answers required to pass. ChatGPT had a lower percentage of correct answers in some areas, such as pharmacology, social welfare, related law and regulations, endocrinology/metabolism, and dermatology, and a higher percentage of correct answers in the areas of nutrition, pathology, hematology, ophthalmology, otolaryngology, dentistry and dental surgery, and nursing integration and practice. Conclusions ChatGPT only passed the 2019 Japanese National Nursing Examination during the most recent 5 years. Although it did not pass the examinations from other years, it performed very close to the passing level, even in those containing questions related to psychology, communication, and nursing.

BACKGROUND The Chat Generative Pre-trained Transformer (ChatGPT), a large language model, has shown good performance on physician certification exams and medical consultations. However, its performance has not been examined in languages other than English or on nursing exams. OBJECTIVE We aimed to evaluate the performance of the ChatGPT on Japanese National Nurse Examinations. METHODS We evaluated the percentage of correct answers provided by the ChatGPT (GPT-3.5) for all questions on the Japanese National Nurse Examination from 2018–2022, excluding inappropriate questions and questions containing images. The exam consists of 240 questions each year, divided into basic knowledge questions that test the basic issues of particular importance to nurses and general questions that test a wide range of specialized knowledge. The format of questions had also two types: simple-choice and situation-setup questions. Simple-choice questions are primarily knowledge-based and multiple-choice, whereas situation-setup questions entail the candidate reading a patient and family situation description, and selecting the nurse's action or patient's response. Hence, the questions were standardized using two types of prompts before requesting answers from the ChatGPT. Chi-square tests were conducted to compare the percentage of correct answers for each year's exam format and specialty area related to the question. In addition, a Cochran-Armitage trend test was performed on the percentage of correct answers from 2018–2022. RESULTS The 5-year average percentage of correct answers for the ChatGPT was 75.1% ± 3.0% for basic knowledge questions and 64.5% ± 5.0% for general questions. The highest percentage of correct answers on the 2018 exam was 80% for basic knowledge questions and 71.2% for general questions. The ChatGPT met the passing criteria for the 2018 Japanese National Nurse Examination and was close to passing the 2019–2022 exams, with only a few more correct answers required to pass. In some areas, such as Pharmacology, Social welfare, Related Law and Regulations, Endocrinology/Metabolism, and Skin, the ChatGPT had lower percentages of correct answers, with higher percentages of correct answers in the areas of Nutrition, Pathology, Hematology, Eye, Ear Nose and Throat, Tooth and Oral, and Nursing Integration and Practice. CONCLUSIONS The ChatGPT only passed the 2018 Japanese National Nursing Examination. Although it did not pass the exams from other years, it performed very close to the passing level, including on psychological, communicational, and nurse-specific questions. CLINICALTRIAL Not applicable.