ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models

Abstract

Purpose This study aimed to assess the performance of ChatGPT, specifically the GPT-3.5 and GPT-4 models, in understanding complex surgical clinical information and its potential implications for surgical education and training. Methods The dataset comprised 280 questions from the Korean general surgery board exams conducted between 2020 and 2022. Both GPT-3.5 and GPT-4 models were evaluated, and their performances were compared using McNemar test. Results GPT-3.5 achieved an overall accuracy of 46.8%, while GPT-4 demonstrated a significant improvement with an overall accuracy of 76.4%, indicating a notable difference in performance between the models (P < 0.001). GPT-4 also exhibited consistent performance across all subspecialties, with accuracy rates ranging from 63.6% to 83.3%. Conclusion ChatGPT, particularly GPT-4, demonstrates a remarkable ability to understand complex surgical clinical information, achieving an accuracy rate of 76.4% on the Korean general surgery board exam. However, it is important to recognize the limitations of large language models and ensure that they are used in conjunction with human expertise and judgment.

Performance of ChatGPT on Nephrology Test Questions

“…Recent studies show GPT-4 outperformed GPT-3.5 by 24%-30% in various medical examinations. 13,14,21,23 These findings indicate a significant enhancement in the model's capabilities. However, a study using the American College of Gastroenterology Test found GPT-3.5 and GPT-4 had scores of 65% and 62%, respectively.…”

“…As a result, questions that involved visual elements, such as clinical images, medical photographs, and graphs, were excluded from our assessment, following the approach taken by previous studies. 7,8,11,13,14…”

Below average ChatGPT performance in medical microbiology exam compared to university students

“…These comparisons highlighted the potential of ChatGPT in higher educational assessments; nevertheless, it showed the importance of ongoing refinements of these models and the dangers of inaccuracies it poses (Lo, 2023;Sallam, 2023;Sallam et al, 2023d;Gill et al, 2024). However, making direct comparisons across variable studies can be challenging due to differences in models implemented, subject fields of the exams, test dates, and the exact approaches of prompt construction (Holmes et al, 2023;Huynh Linda et al, 2023;Meskó, 2023;Oh et al, 2023;Skalidis et al, 2023;Yaa et al, 2023).…”

Artificial intelligence model GPT4 narrowly fails simulated radiological protection exam

“…Questions, along with their multiple-choice answers, were presented to the model followed by the instruction, 'Give the number of the best answer. Start your response with "The answer is:"' The goal of this approach was to have the LLM respond with just the multiple-choice answer (1)(2)(3)(4)(5) and not provide a lengthy (costly) explanation.…”

“…GPT 4.0 has proven remarkable ability in assessing knowledge in specialised domains such as medicine, law, and business [2][3][4]-areas that have historically been the exclusive purview of professionals. Particularly noteworthy is its exceptional performance on assessments like the Korean general surgery board exam, the United States Medical Licensing Exam, and the Wharton MBA final exam, each achieved without the finetuning of the pretrained model [5][6][7].…”