Application and accuracy of artificial intelligence-derived large language models in patients with age related macular degeneration

Abstract

Introduction Age-related macular degeneration (AMD) affects millions of people globally, leading to a surge in online research of putative diagnoses, causing potential misinformation and anxiety in patients and their parents. This study explores the efficacy of artificial intelligence-derived large language models (LLMs) like in addressing AMD patients' questions. Methods ChatGPT 3.5 (2023), Bing AI (2023), and Google Bard (2023) were adopted as LLMs. Patients’ questions were subdivided in two question categories, (a) general medical advice and (b) pre- and post-intravitreal injection advice and classified as (1) accurate and sufficient (2) partially accurate but sufficient and (3) inaccurate and not sufficient. Non-parametric test has been done to compare the means between the 3 LLMs scores and also an analysis of variance and reliability tests were performed among the 3 groups. Results In category a) of questions, the average score was 1.20 (± 0.41) with ChatGPT 3.5, 1.60 (± 0.63) with Bing AI and 1.60 (± 0.73) with Google Bard, showing no significant differences among the 3 groups (p = 0.129). The average score in category b was 1.07 (± 0.27) with ChatGPT 3.5, 1.69 (± 0.63) with Bing AI and 1.38 (± 0.63) with Google Bard, showing a significant difference among the 3 groups (p = 0.0042). Reliability statistics showed Chronbach’s α of 0.237 (range 0.448, 0.096–0.544). Conclusion ChatGPT 3.5 consistently offered the most accurate and satisfactory responses, particularly with technical queries. While LLMs displayed promise in providing precise information about AMD; however, further improvements are needed especially in more technical questions.

Artificial Versus Human Intelligence in the Diagnostic Approach of Ophthalmic Case Scenarios: A Qualitative Evaluation of Performance and Consistency

“…When it comes to comparing different AI chatbots, current literature suggests better performance of ChatGPT than Google Bard/Gemini in triaging and diagnosing simulated ophthalmic patient complaints [ 14 ] or in providing an accurate and coherent surgical plan for glaucoma [ 15 ] and vitreoretinal [ 16 ] cases. Furthermore, ChatGPT-3.5 was found to be more accurate than Bing and Google Bard/Gemini in answering patients’ questions about age-related macular degeneration [ 2 ]. Moreover, in a study evaluating the performance of ChatGPT (versions 3.5 and 4) and Google Bard/Gemini in answering common inquiries regarding ocular symptoms, it was found that ChatGPT-4 outperformed ChatGPT-3.5 and Google Bard; however, all chatbots exhibited only moderate self-awareness capabilities and modest self-improving capabilities over time [ 17 ].…”

“…Despite being developed as conversational artificial intelligence (AI) systems, large language model (LLM)-based chatbots are currently the focus of intense interest in the field of medicine, particularly in ophthalmology. For example, Chat Generative Pre-trained Transformer (ChatGPT) (OpenAI, San Francisco, California, United States), released in late 2022, has already shown remarkable ability in providing general information and advice for glaucoma [ 1 ] and age-related macular degeneration patients [ 2 ], answering ophthalmology StatPearls questions [ 3 ], and even helping triage ophthalmic emergency cases [ 4 ].…”

Applications of artificial intelligence-enabled robots and chatbots in ophthalmology: recent advances and future trends

“…ChatGPT capabilities have been investigated in question comprehending and providing relevant responses as well. A study revealed that ChatGPT delivers responses that are 80% accurate and comprehensive in addressing the 16 most frequently asked questions about AMD, providing valuable information that individuals with AMD would find beneficial [54]. Additionally, another recent study showed the ChatGPT's ability to handle ophthalmic discharge summaries and operative notes.…”