Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios

Abstract

ObjectiveTo evaluate the clinical applications and limitations of chat generative pretrained transformer (ChatGPT) in otolaryngology.Study DesignCross‐sectional survey.SettingTertiary academic center.MethodsChatGPT 4.0 was queried for diagnoses and management plans for 20 physician‐written clinical vignettes in otolaryngology. Attending physicians were then asked to rate the difficulty of the clinical vignettes and agreement with the differential diagnoses and management plans of ChatGPT responses on a 5‐point Likert scale. Summary statistics were calculated. Univariate ordinal regression was then performed between vignette difficulty and quality of the diagnoses and management plans.ResultsEleven attending physicians completed the survey (61% response rate). Overall, vignettes were rated as very easy to neutral difficulty (range of median score: 1.00‐4.00; overall median 2.00). There was a high agreement with the differential diagnosis provided by ChatGPT (range of median score: 3.00‐5.00; overall median: 5.00). There was also high agreement with treatment plans (range of median score: 3.00‐5.00; overall median: 5.00). There was no association between vignette difficulty and agreement with differential diagnosis or treatment. Lower diagnosis scores had greater odds of having lower treatment scores.ConclusionGenerative artificial intelligence models like ChatGPT are being rapidly adopted in medicine. Performance with curated, easy‐to‐moderate difficulty otolaryngology scenarios indicate high agreement with physicians for diagnosis and management. However, a decreased quality in diagnosis is associated with decreased quality in management. Further research is necessary on ChatGPT's ability to handle unstructured clinical information.

ChatGPT vs UpToDate: comparative study of usefulness and reliability of Chatbot in common clinical presentations of otorhinolaryngology–head and neck surgery

“…Brennan et al reported ChatGPT benefit on ear, nose, and throat (ENT) surgical education [ 30 ]. Qu et al evaluated the diagnostic application of ChatGPT and reported the low quality of the Chatbot [ 4 ]. Hoch et al assessed ChatGPT skills in single and multiple choice ENT board questions, and it performed a low correct answer percentage [ 8 ].…”

ChatGPT Guided Diagnosis of Ameloblastic Fibro-Odontoma: A Case Report with Eventful Healing

“…At this point, the AI evaluating these pathological conditions has major advantages over the human evaluators. For instance, the human brain, limited in data storage, tends to perceive rare medical conditions and treatments as more challenging [1]. AI has no such limitations and can instantly access countless data sources, such as articles and reviews, deriving conclusions from their content [3].…”

“…The use of artificial intelligence (AI) in medicine has shown promising results in improving diagnostic accuracy and streamlining the decision-making process [1]. One of them, Chat Generative Pre-Trained Transformer (ChatGPT), released to the public on November 30, 2022, is the largest artificial intelligence language model and a prototype developed by OpenAI, specializing in dialogue [2].…”

“…The main advantages are that it reduces human error, provides assistance and guidance to doctors and other healthcare professionals, and is always accessible for patient services [4]. This tool holds the potential to function as a valuable aid and guide for clinicians, particularly in rare conditions [1]. Ameloblastic Fibro-Odontoma (AFO) is a uncommon benign odontogenic tumor, sharing similar histologic features with Ameloblastic Fibroma (AF) [5].…”

CanChatGPThelp patients answer their otolaryngology questions?

“…As Board‐certified otolaryngologists, we agreed with the ChatGPT response 98.3%, consistent with recent Otolaryngology literature regarding accuracy of AI in response to patient vignettes. 7 In surgery adjacent fields, ChatGPT has also performed at the accuracy level of a resident in response to self‐assessment and in‐service training questions in the fields of plastic surgery and neurosurgery, respectively. 8 , 9 …”