Performance of ChatGPT on free-response, clinical reasoning exams

Abstract

Importance: Studies show that ChatGPT, a general purpose large language model chatbot, could pass the multiple-choice US Medical Licensing Exams, but the model's performance on open-ended clinical reasoning is unknown. Objective: To determine if ChatGPT is capable of consistently meeting the passing threshold on free-response, case-based clinical reasoning assessments. Design: Fourteen multi-part cases were selected from clinical reasoning exams administered to pre-clerkship medical students between 2019 and 2022. For each case, the questions were run through ChatGPT twice and responses were recorded. Two clinician educators independently graded each run according to a standardized grading rubric. To further assess the degree of variation in ChatGPT's performance, we repeated the analysis on a single high-complexity case 20 times. Setting: A single US medical school Participants: ChatGPT Main Outcomes and Measures: Passing rate of ChatGPT's scored responses and the range in model performance across multiple run throughs of a single case. Results: 12 out of the 28 ChatGPT exam responses achieved a passing score (43%) with a mean score of 69% (95% CI: 65% to 73%) compared to the established passing threshold of 70%. When given the same case 20 separate times, ChatGPT's performance on that case varied with scores ranging from 56% to 81%. Conclusions and Relevance: ChatGPT's ability to achieve a passing performance in nearly half of the cases analyzed demonstrates the need to revise clinical reasoning assessments and incorporate artificial intelligence (AI)-related topics into medical curricula and practice.

Accuracy of ChatGPT‐Generated Information on Head and Neck and Oromaxillofacial Surgery: A Multicenter Collaborative Analysis

“…This could lead to faster and more accurate diagnoses, improving patient outcomes. [3][4][5] Second, ChatGPT can help synthesize and analyze vast amounts of medical literature, which could lead to the discovery of new treatments, medications, or a better understanding of diseases. The potential of ChatGPT in medical writing is currently the most studied and discussed in the literature.…”

A User-friendly Approach for the Diagnosis of Diabetic Retinopathy Using ChatGPT and Automated Machine Learning

“…This indicates a significant degree of variability in ChatGPT's responses, even when faced with identical scenarios. 42 Another study aimed to evaluate ChatGPT's capacity for ongoing clinical decision support. The research involved inputting published clinical vignettes into ChatGPT-3.5 and assessing its accuracy in various areas such as differential diagnoses, diagnostic testing, final diagnosis, and management.…”

Reliability of large language models in managing odontogenic sinusitis clinical scenarios: a preliminary multidisciplinary evaluation

“…Yet, any evaluation of these tools must be context-specific and rigorous. This assessment is particularly relevant, urgent, and novel for complex conditions, such as ODS, where optimal multidisciplinary diagnostic and therapeutic paradigms are challenging to establish due to a relatively scarce body of recently published evidence [ 11 ]. This study aimed to evaluate the reliability of two versions of the ChatGPT LLM in managing ODS.…”