WITHDRAWN: ChatGPT-4 as a Board-Certified Surgeon: A Pilot Study

Abstract

PurposeLarge language models (LLMs), such as GPT-4 (OpenAI; San Francisco, CA), are promising tools for surgical education. However, skepticism about their accuracy and reliability remains a significant barrier to their widespread adoption. Although GPT-4 has demonstrated a remarkable ability to pass multiple-choice tests, its general surgery knowledge and clinical judgment in complex oral-based examinations are less clear. This study aims to evaluate GPT-4’s general surgery knowledge using written and oral board-style examinations to drive improvements that will enable the tool to revolutionize surgical education and practice.MethodsWe tested GPT-4’s ability to answer 250 random multiple-choice questions (MCQs) from the Surgical Council on Resident Education (SCORE) question bank and navigate four oral board scenarios derived from the Entrustable Professional Activities (EPA) topic list. Two former oral board examiners assessed the responses independently for accuracy.ResultsOn MCQs, GPT-4 answered 197 out of 250 (78.8%) correctly, corresponding to a 99% probability of passing the American Board of Surgery Qualifying Examination (ABS QE). On oral board scenarios, GPT-4 committed critical failures in three of four (75%) clinical cases. Common reasons for failure were incorrect timing of intervention and incorrect suggested operation.ConclusionsWhile GPT-4’s high performance on MCQs mirrored prior studies, the model struggled to generate accurate long-form content in our mock oral board examination. Future efforts should use specialized datasets and advanced reinforcement learning to enhance GPT-4’s contextual understanding and clinical judgment.