Comparison of the problem-solving performance of ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard for the Korean emergency medicine board examination question bank

Abstract

Large language models (LLMs) have been deployed in diverse fields, and the potential for their application in medicine has been explored through numerous studies. This study aimed to evaluate and compare the performance of ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard for the Emergency Medicine Board Examination question bank in the Korean language. Of the 2353 questions in the question bank, 150 questions were randomly selected, and 27 containing figures were excluded. Questions that required abilities such as analysis, creative thinking, evaluation, and synthesis were classified as higher-order questions, and those that required only recall, memory, and factual information in response were classified as lower-order questions. The answers and explanations obtained by inputting the 123 questions into the LLMs were analyzed and compared. ChatGPT-4 (75.6%) and Bing Chat (70.7%) showed higher correct response rates than ChatGPT-3.5 (56.9%) and Bard (51.2%). ChatGPT-4 showed the highest correct response rate for the higher-order questions at 76.5%, and Bard and Bing Chat showed the highest rate for the lower-order questions at 71.4%. The appropriateness of the explanation for the answer was significantly higher for ChatGPT-4 and Bing Chat than for ChatGPT-3.5 and Bard (75.6%, 68.3%, 52.8%, and 50.4%, respectively). ChatGPT-4 and Bing Chat outperformed ChatGPT-3.5 and Bard in answering a random selection of Emergency Medicine Board Examination questions in the Korean language.

An Evaluation of General-Purpose AI Chatbots: A Comprehensive Comparative Analysis

“…All authors only compared the most popular chatbots, which, in most cases (except ChatGPT-4.0 and Bing Chat), use different language models. In four out of five articles, authors indicated that Bard received the lowest score [22,23,25,26]. Only in one article related to medicine did Bard receive the highest score [24].…”

“…Additionally, in Plevris et al, (2023) study [22], authors indicated that giving Internet access to the chatbot can provide more precise answers, particularly if these questions are available in open access. Furthermore, it was found that ChatGPT-4.0, using its LLM model, was more accurate than Bing Chat [23].…”

“…This highlights the importance for developers to configure the search option correctly for their chatbots. Lee et al, in their article [23], indicated that the percentage of incorrect answers in ChatGPT-3.5 and Bard are almost similar and consist of almost 45% of wrong answers, while BingChat had only 23% of wrong answers. Salazar et al, in their article [24], compared ChatGPT, Bing Chat, and Bard.…”