On the limitations of large language models in clinical diagnosis

Abstract

Background: The potential of large language models (LLM) such as GPT to support complex tasks such as differential diagnosis has been a subject of debate, with some ascribing near sentient abilities to the models and others claiming that LLMs merely perform "autocomplete on steroids". A recent study reported that the Generative Pretrained Transformer 4 (GPT-4) model performed well in complex differential diagnostic reasoning. The authors assessed the performance of GPT-4 in identifying the correct diagnosis in a series of case records from the New England Journal of Medicine. The authors constructed prompts based on the clinical presentation section of the case reports, and compared the results of GPT-4 to the actual diagnosis. GPT-4 returned the correct diagnosis as a part of its response in 64% of cases, with the correct diagnosis being at rank 1 in 39% of cases. However, such concise but comprehensive narratives of the clinical course are not typically available in electronic health records (EHRs). Further, if they were available, EHR records contain identifying information whose transmission is prohibited by Health Insurance Portability and Accountability Act (HIPAA) regulations. Methods: To assess the expected performance of GPT on comparable datasets that can be generated by text mining and by design cannot contain identifiable information, we parsed the texts of the case reports and extracted Human Phenotype Ontology (HPO) terms, from which prompts for GPT were constructed that contain largely the same clinical abnormalities but lack the surrounding narrative. Results: While the performance of GPT-4 on the original narrative-based text was good, with the final diagnosis being included in its differential in 29/75 cases (38.7%; rank 1 in 17.3% of cases; mean rank of 3.4), the performance of GPT-4 on the feature-based approach that includes the major clinical abnormalities without additional narrative texas substantially worse, with GPT-4 including the final diagnosis in its differential in 8/75 cases (10.7%; rank 1 in 4.0% of cases; mean rank of 3.9). Interpretation: We consider the feature-based queries to be a more appropriate test of the performance of GPT-4 in diagnostic tasks, since it is unlikely that the narrative approach can be used in actual clinical practice. Future research and algorithmic development is needed to determine the optimal approach to leveraging LLMs for clinical diagnosis.

Integrating Retrieval-Augmented Generation with Large Language Models in Nephrology: Advancing Practical Applications

“…For example, some investigators have shown that they can offer tailored medical guidance [ 12 ], distribute educational resources [ 7 ], and improve the quality of medical training [ 7 , 13 , 14 ]. These tools can also support clinical decision making [ 15 , 16 , 17 ], help identify urgent medical situations [ 18 ], and respond to patient inquiries with understanding and empathy [ 19 , 20 , 21 ]. Extensive research has shown that ChatGPT, particularly its most recent version GPT-4, excels across various standardized tests.…”

Just how transformative will AI/ML be for immuno-oncology?

“…This is because currently interactions with ChatGPT are transmitted over the internet. 83 Additionally, if limited to basic (non-identifying) diagnostic information, the performance of ChatGPT degrades considerably. 83 These and related issues have influenced the development of LLMs derived specifically from deidentified clinical notes as well as other relevant sources.…”

“… 83 Additionally, if limited to basic (non-identifying) diagnostic information, the performance of ChatGPT degrades considerably. 83 These and related issues have influenced the development of LLMs derived specifically from deidentified clinical notes as well as other relevant sources. 82 Despite general advances in LLM capabilities, the robust application to biomedical and clinical text is likely a mid-term beneficiary of AI/ML advancements.…”

A critical review of large language models: Sensitivity, bias, and the path toward specialized AI

“…While these models are robust in representing the behavior of the training data, they are lighter and easier to diagnose compared to LLMs. Concerns around the interpretability and potential biases of large black-box models like LLMs could make simpler, more transparent ML models preferable in high-stakes domains like healthcare diagnosis, where clinician oversight and autonomy are important (Reese et al, 2023;Fisch, 2024;Κούρου et al, 2015;Kavakiotis et al, 2017;Salvatore et al, 2014;Malek, 2023;Zack et al, 2023;Abu-Jeyyab, 2023;Huang et al, 2023;Takahashi, 2023). Lehman et al (2023) explore whether LLMs, trained primarily with general web text, are suitable for specialized domains like clinical text.…”