Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination

Abstract

Background: Artificial intelligence (AI) holds potential in improving medical education and healthcare delivery. ChatGPT is a state-of-the-art natural language processing AI model which has shown impressive capabilities, scoring in the top percentiles on numerous standardized examinations, including the Uniform Bar Exam and Scholastic Aptitude Test. The goal of this study was to evaluate ChatGPT performance on the Orthopaedic In-Training Examination (OITE), an assessment of medical knowledge for orthopedic residents. Methods: OITE 2020, 2021, and 2022 questions without images were inputted into ChatGPT version 3.5 and version 4 (GPT-4) with zero prompting. The performance of ChatGPT was evaluated as a percentage of correct responses and compared with the national average of orthopedic surgery residents at each postgraduate year (PGY) level. ChatGPT was asked to provide a source for its answer, which was categorized as being a journal article, book, or website, and if the source could be verified. Impact factor for the journal cited was also recorded. Results: ChatGPT answered 196 of 360 answers correctly (54.3%), corresponding to a PGY-1 level. ChatGPT cited a verifiable source in 47.2% of questions, with an average median journal impact factor of 5.4. GPT-4 answered 265 of 360 questions correctly (73.6%), corresponding to the average performance of a PGY-5 and exceeding the corresponding passing score for the American Board of Orthopaedic Surgery Part I Examination of 67%. GPT-4 cited a verifiable source in 87.9% of questions, with an average median journal impact factor of 5.2. Conclusions: ChatGPT performed above the average PGY-1 level and GPT-4 performed better than the average PGY-5 level, showing major improvement. Further investigation is needed to determine how successive versions of ChatGPT would perform and how to optimize this technology to improve medical education. Clinical Relevance: AI has the potential to aid in medical education and healthcare delivery.

Conformity of ChatGPT recommendations with the AUA/SUFU guideline on postprostatectomy urinary incontinence

“…This is particularly alarming considering that the access is free for this version, which is the most used. The substantial increase in performance of version 4 is consistent with other studies 5,[7][8][9]15 and likely stems from the foundational differences in training and algorithmic sophistication between the two versions. 1 It may partially be attributed to the integration of a rule-based reward model in version 4.…”

“…Several studies have assessed the reliability of ChatGPT, each using different methodologies and subsequently reporting varying levels of performance 5–11,15 . Studies using common questions of interest to patients and laypeople usually show high accuracy rates.…”

“…Several studies have assessed the reliability of ChatGPT, each using different methodologies and subsequently reporting varying levels of performance. [5][6][7][8][9][10][11]15 Studies using common questions of interest to patients and laypeople usually show high accuracy rates. In a recent study evaluating the chatbot recommendations regarding breast cancer prevention and screening, an accuracy rate of 88% for ChatGPT 3.5 was reported.…”

“…This tool has shown great promise in aiding healthcare professionals by providing quick and accessible information, assisting in decision‐making processes, and improving patient care 2 . The use of ChatGPT in clinical practice by both physicians and patients has been subject of several studies since its public release in 2022 2–11 . The advent of ChatGPT has marked a significant advancement in the application of AI in the medical field, where rapid access to accurate information can be critical.…”

“…2 The use of ChatGPT in clinical practice by both physicians and patients has been subject of several studies since its public release in 2022. [2][3][4][5][6][7][8][9][10][11] The advent of ChatGPT has marked a significant advancement in the application of AI in the medical field, where rapid access to accurate information can be critical.…”