Accuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images

Mihalache, Andrew; Huang, Ryan S.; Popovic, Marko M.; Patil, Nikhil S.; Pandya, Bhadra U.; Shor, Reut; Pereira, Austin; Kwok, Jason M.; Yan, Peng; Wong, David T.; Kertes, Peter J.; Muni, Rajeev H.

doi:10.1001/jamaophthalmol.2024.0017

“…These findings, along with those from the study by Mihalache et al, illustrate the current limitations of LLMs in the interpretation of medical imaging studies, indicating a substantial gap between its potential and current capabilities in medical diagnostics. While these AI models hold promise for revolutionizing various aspects of clinical practice, their accuracy in image-based diagnostics is currently limited by their inherent design for visual data processing.…”

supporting

confidence: 52%

“…To the Editor We read with interest the article “Accuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images” by Mihalache et al in which the authors continued their exploration into the utility of ChatGPT-4 (OpenAI), an advanced artificial intelligence (AI) chatbot, in ophthalmology . Their latest study critically evaluated the ability of this chatbot to answer multiple-choice questions in ophthalmology, focusing on its performance in interpreting ophthalmic images.…”

mentioning

confidence: 99%

Challenges of Integrating Chatbot Use in Ophthalmology Diagnostics

Koga,

Du

2024

JAMA Ophthalmol

0

View full text Add to dashboard Cite

Oke et al requested the percentage of ophthalmology referrals, and we included these in our article in Figure 1B and in the Results section. Rates of referral increased from 17.0% (1279 of 7505) in 2015 to 23.6% (1836 of 7792) in 2018, then decreased to 15.7% (997 of 6354) in 2022. 1 Regarding completed referrals, we found a similar trend: 7.2% (543 of 7505) in 2015, increasing to 11% (855 of 7792) in 2018, and then decreasing to 6.7% (426 of 6354) in 2022. As posited by Oke et al, we observed in the completed referrals similar racial and ethnic disparities as reported in the article 1 for referral orders, with Asian individuals having the highest completion and White individuals having the lowest completion from 2015 to 2022 for both referral orders (Asian, 23.6% [4531 of 19 164]; Black, 17.6% [59 of 335]; Hispanic, 18% [1094 of 6067]; and White, 14.9% [2026 of 13 613]; P < .001) and completed referrals (Asian, 10% [1921 of 19 164]; Black, 9% [30 of 335]; Hispanic, 7.5% [456 of 6067]; and White, 7.5% [1015 of 13 613]; P < .001) (race and ethnicity data were obtained from the electronic health record, which listed the following options: Asian, Black, Hispanic, White, other [American Indian, multiracial, Native Hawaiian, and other race not further specified], and unknown).We agree it is interesting that there has been a decrease in amblyopia diagnoses, and to answer the question from Oke et al, the diagnoses were made by both primary care and specialists during the study time frame. Given that there may have been some additional well-child care visits toward the end of 2022, there may be some missed amblyopia diagnoses as the patients may not have had sufficient follow-up time in which to observe a subsequent appointment and diagnosis.We concur that future analyses should use some more quasi-experimental methods like difference-in-differences or interrupted time series to further evaluate the impact of interventions. It is important to follow children up over longer periods and to identify those at risk of preventable vision loss.

show abstract

“…The significantly higher accuracy (70%) achieved by Mihalache A et al using GPT-4V on OCT images might be attributed to their use of multiple-choice questions, while our study used open-ended questions. [22]…”

Section: Discussionmentioning

confidence: 99%

Unveiling the Clinical Incapabilities: A Benchmarking Study of GPT-4V(ision) for Ophthalmic Multimodal Image Analysis

Xu,

Chen,

Zhao

et al. 2023

Preprint

View full text Add to dashboard Cite

BackgroundsGPT4-V(ision) has generated great interest across various fields, while its performance in ocular multimodal images is still unknown. This study aims to evaluate the capabilities of a GPT-4V-based chatbot in addressing queries related to ocular multimodal images.MethodsA digital ophthalmologist app was built based on GPT-4V. The evaluation dataset comprised various ocular imaging modalities: slit-lamp, scanning laser ophthalmoscopy (SLO), fundus photography of the posterior pole (FPP), optical coherence tomography (OCT), fundus fluorescein angiography (FFA), and ocular ultrasound (OUS). Each modality included images representing 5 common and 5 rare diseases. The chatbot was presented with ten questions per image, focusing on examination identification, lesion detection, diagnosis, decision support, and the repeatability of diagnosis. The responses of GPT-4V were evaluated based on accuracy, usability, and safety.ResultsThere was a substantial agreement among three ophthalmologists. Out of 600 responses, 30.5% were accurate, 22.8% of 540 responses were highly usable, and 55.5% of 540 responses were considered safe by ophthalmologists. The chatbot excelled in interpreting slit-lamp images, with 42.0%, 42.2%, and 68.5% of the responses being accurate, highly usable, and no harm, respectively. However, its performance was notably weaker in FPP images, with only 13.7%, 3.7%, and 38.5% in the same categories. It correctly identified 95.6% of the imaging modalities. For lesion identification, diagnosis, and decision support, the chatbot’s accuracy was 25.6%, 16.1%, and 24.0%, respectively. The average proportions of correct answers, highly usable, and no harm for GPT-4V in common diseases were 37.9%, 30.5%, and 60.1%, respectively. These proportions were all higher compared to those in rare diseases, which were 23.2% (P<0.001), 15.2% (P<0.001), and 51.1% (P=0.032), respectively. The overall repeatability of GPT4-V in diagnosing ocular images was 63% (38/60).ConclusionCurrently, GPT-4V lacks the reliability required for clinical decision-making and patient consultation in ophthalmology. Ongoing refinement and testing are essential for improving the efficacy of large language models in medical applications.

show abstract

“…In Reply We sincerely thank Koga and Du for their comments on our article . As discussed by Koga and Du, we note that large language models, such as ChatGPT (OpenAI), are currently limited in the domain of ophthalmic imaging interpretation.…”

mentioning

confidence: 99%

“…As discussed by Koga and Du, we note that large language models, such as ChatGPT (OpenAI), are currently limited in the domain of ophthalmic imaging interpretation. Our recent investigation found that the chatbot underperformed on image-based questions compared with questions that did not directly require ophthalmic imaging interpretation . As per the ChatGPT-4V(ision) system card, OpenAI notes that the chatbot’s imaging interpretation features are unsuitable for carrying out any medical function.…”

mentioning

confidence: 99%

Challenges of Integrating Chatbot Use in Ophthalmology Diagnostics—Reply

Mihalache,

Popovic,

Muni

2024

JAMA Ophthalmol

0

View full text Add to dashboard Cite

Oke et al requested the percentage of ophthalmology referrals, and we included these in our article in Figure 1B and in the Results section. Rates of referral increased from 17.0% (1279 of 7505) in 2015 to 23.6% (1836 of 7792) in 2018, then decreased to 15.7% (997 of 6354) in 2022. 1 Regarding completed referrals, we found a similar trend: 7.2% (543 of 7505) in 2015, increasing to 11% (855 of 7792) in 2018, and then decreasing to 6.7% (426 of 6354) in 2022. As posited by Oke et al, we observed in the completed referrals similar racial and ethnic disparities as reported in the article 1 for referral orders, with Asian individuals having the highest completion and White individuals having the lowest completion from 2015 to 2022 for both referral orders (Asian, 23.6% [4531 of 19 164]; Black, 17.6% [59 of 335]; Hispanic, 18% [1094 of 6067]; and White, 14.9% [2026 of 13 613]; P < .001) and completed referrals (Asian, 10% [1921 of 19 164]; Black, 9% [30 of 335]; Hispanic, 7.5% [456 of 6067]; and White, 7.5% [1015 of 13 613]; P < .001) (race and ethnicity data were obtained from the electronic health record, which listed the following options: Asian, Black, Hispanic, White, other [American Indian, multiracial, Native Hawaiian, and other race not further specified], and unknown).We agree it is interesting that there has been a decrease in amblyopia diagnoses, and to answer the question from Oke et al, the diagnoses were made by both primary care and specialists during the study time frame. Given that there may have been some additional well-child care visits toward the end of 2022, there may be some missed amblyopia diagnoses as the patients may not have had sufficient follow-up time in which to observe a subsequent appointment and diagnosis.We concur that future analyses should use some more quasi-experimental methods like difference-in-differences or interrupted time series to further evaluate the impact of interventions. It is important to follow children up over longer periods and to identify those at risk of preventable vision loss.

show abstract

Accuracy of an Artificial Intelligence Chatbot’s Interpretation of Clinical Ophthalmic Images

Cited by 33 publications

References 28 publications

Challenges of Integrating Chatbot Use in Ophthalmology Diagnostics

Challenges of Integrating Chatbot Use in Ophthalmology Diagnostics

Unveiling the Clinical Incapabilities: A Benchmarking Study of GPT-4V(ision) for Ophthalmic Multimodal Image Analysis

Challenges of Integrating Chatbot Use in Ophthalmology Diagnostics—Reply

Contact Info

Product

Resources

About