When vision meets reality: Exploring the clinical applicability of GPT-4 with vision

Deng, Jiawen; Heybati, Kiyan; Shammas-Toma, Matthew

doi:10.1016/j.clinimag.2024.110101

Cited by 16 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is in line with several prior studies reporting low diagnostic performance of GPT-4(V) with radiological images alone as input (12,18,20,21). Consistent with the findings of Schubert et al, we observed that the combination of medical history and radiological images yielded higher diagnostic accuracy than images alone, although the difference did not reach statistical significance (17).…”

Section: Discussionsupporting

confidence: 92%

“…For the binary scoring system, a chi-square test was performed over all groups with subsequent pairwise testing. For the numeric scoring system, a Kruskal-Wallis test was conducted across all groups followed by pairwise testing using Dunn’s test (21). For both scoring systems, results were adjusted to a false-discovery rate of 0.05 by employing the Benjamini-Hochberg procedure (22).…”

Section: Methodsmentioning

confidence: 99%

“…However, several studies demonstrated low performance of Generative Pre-trained Transformer 4 Vision (GPT-4V) by OpenAI in differential diagnosis based on various types of radiological images (12,16,18,20,21). These studies evaluated queries with only images (12,18,20,21) or clinical information and images (16) as input.…”

Section: Introductionmentioning

confidence: 99%

“…Recent studies have further explored the diagnostic application of multimodal LLMs (also called 'vision-language models') that are able to ingest not only text but also image data as input (12)(13)(14)(15)(16)(17)(18)(19)(20). However, several studies demonstrated low performance of Generative Pretrained Transformer 4 Vision (GPT-4V) by OpenAI in differential diagnosis based on various types of radiological images (12,16,18,20,21). These studies evaluated queries with only images (12,18,20,21) or clinical information and images (16) as input.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Impact of Multimodal Prompt Elements on Diagnostic Performance of GPT-4(V) in Challenging Brain MRI Cases

Schramm,

Preis,

Metz

et al. 2024

Preprint

View full text Add to dashboard Cite

Background Recent studies have explored the application of multimodal large language models (LLMs) in radiological differential diagnosis. Yet, how different multimodal input combinations affect diagnostic performance is not well understood. Purpose To evaluate the impact of varying multimodal input elements on the accuracy of GPT-4(V)-based brain MRI differential diagnosis. Methods Thirty brain MRI cases with a challenging yet verified diagnosis were selected. Seven prompt groups with variations of four input elements (image, image annotation, medical history, image description) were defined. For each MRI case and prompt group, three identical queries were performed using an LLM-based search engine (© PerplexityAI, powered by GPT-4(V)). Accuracy of LLM-generated differential diagnoses was rated using a binary and a numeric scoring system and analyzed using a chi-square test and a Kruskal-Wallis test. Results were corrected for false discovery rate employing the Benjamini-Hochberg procedure. Regression analyses were performed to determine the contribution of each individual input element to diagnostic performance. Results The prompt group containing an annotated image, medical history, and image description as input exhibited the highest diagnostic accuracy (67.8% correct responses). Significant differences were observed between prompt groups, especially between groups that contained the image description among their inputs, and those that did not. Regression analyses confirmed a large positive effect of the image description on diagnostic accuracy (p << 0.001), as well as a moderate positive effect of the medical history (p < 0.001). The presence of unannotated or annotated images had only minor or insignificant effects on diagnostic accuracy. Conclusion The textual description of radiological image findings was identified as the strongest contributor to performance of GPT-4(V) in brain MRI differential diagnosis, followed by the medical history. The unannotated or annotated image alone yielded very low diagnostic performance. These findings offer guidance on the effective utilization of multimodal LLMs in clinical practice.

show abstract

Section: Discussionsupporting

confidence: 92%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Impact of Multimodal Prompt Elements on Diagnostic Performance of GPT-4(V) in Challenging Brain MRI Cases

Schramm,

Preis,

Metz

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Similar to hallucination rates described previously (25), we found statements inconsistent with widely accepted medical knowledge in 5.1% of LLM responses. Many of these involved incorrect interpretations of MRI screenshots provided as input, confirming earlier studies demonstrating low performance of current state-of-the art LLMs in diagnostic tasks based on radiological images (27–30). Interestingly, hallucinations were even found with PerplexityAI, which – unlike other Chatbots such as ChatGPT - combines LLMs with real-time information retrieval from the internet to support its answers with relevant sources (31).…”

Section: Discussionsupporting

confidence: 77%

Boosting LLM-Assisted Diagnosis: 10-Minute LLM Tutorial Elevates Radiology Residents’ Performance in Brain MRI Interpretation

Kim,

Schramm,

Wihl

et al. 2024

Preprint

View full text Add to dashboard Cite

PurposeTo evaluate the impact of a structured tutorial on the use of a large language model (LLM)-based search engine on radiology residents’ performance in LLM-assisted brain MRI differential diagnosis.Materials & MethodsIn this retrospective study, nine radiology residents determined the three most likely differential diagnoses for three sets of ten brain MRI cases with a challenging yet definite diagnosis. Each set of cases was assessed 1) with the support of conventional internet search, 2) using an LLM-based search engine (© Perplexity AI) without prior training, or 3) with LLM assistance after a structured 10-minute tutorial on how to effectively use the tool for differential diagnosis. The tutorial content was based on the results of two studies on LLM-assisted radiological diagnosis and included a prompt template. Reader responses were rated using a binary and numeric scoring system. Reading times were tracked and confidence levels were recorded on a 5-point Likert scale. Binary and numeric scores were analyzed using chi-square tests and pairwise Mann-Whitney U tests each. Search engine logs were examined to quantify user interaction metrics, and to identify hallucinations and misinterpretations in LLM responses.ResultsRadiology residents achieved the highest accuracy when employing the LLM-based search engine following the tutorial, indicating the correct diagnosis among the top three differential diagnoses in 62.5% of cases (55/88). This was followed by the LLM-assisted workflow before the tutorial (44.8%; 39/87) and the conventional internet search workflow (32.2%; 28/87). The LLM tutorial led to significantly higher performance (binary scores: p = 0.042, numeric scores: p = 0.016) and confidence (p = 0.006) but resulted in no relevant differences in reading times. Hallucinations were found in 5.1% of LLM queries.ConclusionA structured 10-minute LLM tutorial increased performance and confidence levels in LLM-assisted brain MRI differential diagnosis among radiology residents.Clinical Relevance StatementOur findings highlight the considerable benefits that even low-cost, low-effort educational interventions on LLMs can provide. Integrating LLM education in radiology training programs could augment practitioners’ capacity to harness AI technologies effectively.

show abstract

ChatGPT’s diagnostic performance based on textual vs. visual information compared to radiologists’ diagnostic performance in musculoskeletal radiology

Horiuchi,

Tatekawa,

Oura

et al. 2024

Eur Radiol

View full text Add to dashboard Cite

Objectives To compare the diagnostic accuracy of Generative Pre-trained Transformer (GPT)-4-based ChatGPT, GPT-4 with vision (GPT-4V) based ChatGPT, and radiologists in musculoskeletal radiology. Materials and methods We included 106 “Test Yourself” cases from Skeletal Radiology between January 2014 and September 2023. We input the medical history and imaging findings into GPT-4-based ChatGPT and the medical history and images into GPT-4V-based ChatGPT, then both generated a diagnosis for each case. Two radiologists (a radiology resident and a board-certified radiologist) independently provided diagnoses for all cases. The diagnostic accuracy rates were determined based on the published ground truth. Chi-square tests were performed to compare the diagnostic accuracy of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and radiologists. Results GPT-4-based ChatGPT significantly outperformed GPT-4V-based ChatGPT (p < 0.001) with accuracy rates of 43% (46/106) and 8% (9/106), respectively. The radiology resident and the board-certified radiologist achieved accuracy rates of 41% (43/106) and 53% (56/106). The diagnostic accuracy of GPT-4-based ChatGPT was comparable to that of the radiology resident, but was lower than that of the board-certified radiologist although the differences were not significant (p = 0.78 and 0.22, respectively). The diagnostic accuracy of GPT-4V-based ChatGPT was significantly lower than those of both radiologists (p < 0.001 and < 0.001, respectively). Conclusion GPT-4-based ChatGPT demonstrated significantly higher diagnostic accuracy than GPT-4V-based ChatGPT. While GPT-4-based ChatGPT’s diagnostic performance was comparable to radiology residents, it did not reach the performance level of board-certified radiologists in musculoskeletal radiology. Clinical relevance statement GPT-4-based ChatGPT outperformed GPT-4V-based ChatGPT and was comparable to radiology residents, but it did not reach the level of board-certified radiologists in musculoskeletal radiology. Radiologists should comprehend ChatGPT’s current performance as a diagnostic tool for optimal utilization. Key Points This study compared the diagnostic performance of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and radiologists in musculoskeletal radiology. GPT-4-based ChatGPT was comparable to radiology residents, but did not reach the level of board-certified radiologists. When utilizing ChatGPT, it is crucial to input appropriate descriptions of imaging findings rather than the images. Graphical Abstract

show abstract

When vision meets reality: Exploring the clinical applicability of GPT-4 with vision

Cited by 16 publications

References 15 publications

Impact of Multimodal Prompt Elements on Diagnostic Performance of GPT-4(V) in Challenging Brain MRI Cases

Impact of Multimodal Prompt Elements on Diagnostic Performance of GPT-4(V) in Challenging Brain MRI Cases

Boosting LLM-Assisted Diagnosis: 10-Minute LLM Tutorial Elevates Radiology Residents’ Performance in Brain MRI Interpretation

ChatGPT’s diagnostic performance based on textual vs. visual information compared to radiologists’ diagnostic performance in musculoskeletal radiology

Contact Info

Product

Resources

About