Accuracy of ChatGPT on Medical Questions in the National Medical Licensing Examination in Japan: Evaluation Study (Preprint)

Abstract

UNSTRUCTURED ChatGPT (Open AI, San Francisco, California, USA) has gained considerable attention because of its natural and intuitive responses. One limitation of OpenAI is its failure to perform reinforcement learning based on reliable information, thereby providing inaccurate or meaningless answers. Fortunately, on March 2023 update introduced GPT-4, which, according to internal evaluations, is expected to increase the likelihood of producing factual responses by 40% compared with its predecessor, GPT-3.5. We verified the accuracy of ChatGPT based on GPT-4 (ChatGPT4) and based on GPT-3.5 (ChatGPT3.5) by solving the Japanese National Medical Examination. We excluded questions containing figures and tables unsupported by ChatGPT. Of the 400 questions, 292 were analyzed. The correct response rate for ChatGPT4 was 81.5%, which was significantly higher than 42.8%, the rate for ChatGPT3.5. Moreover, ChatGPT4 surpassed the passing standard (>72%) for the Japanese National Medical Examination, indicating its potential as a diagnostic and therapeutic decision aid for physicians. We anticipate that future updates of ChatGPT will further enhance its accuracy, making it an invaluable resource in the field of medicine.

Will Artificial Intelligence be Useful (or Misused) in Environmental Toxicology and Chemistry?

“…A simple example is the little pop‐up “chat” window that appears when accessing information or asking a question online from an organization. However, since November 2022 a much more powerful chatbot for writing and other tasks was posted as an open‐access/free program—it is called “ChatGPT” (Vanian, 2022). Over the past few months, ChatGPT has become popular among college students, seeking a way to satisfy writing assignments in an expedited manner.…”