New meaning for NLP: the trials and tribulations of natural language processing with GPT-3 in ophthalmology

Nath, Siddharth; Marie, Abdullah; Ellershaw, Simon; Korot, Edward; Keane, Pearse A.

doi:10.1136/bjophthalmol-2022-321141

Cited by 90 publications

(43 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the past months, there has been significant interest in examining the utility of LLMs in medicine 5 . Despite having encouraging impacts in various industries, it is important to thoroughly evaluate their performance and biases before determining their clinical usefulness 4 19 . In this study, we provide evidence on the performance of ChatGPT, a non-domain specific LLM, in responding to questions found on the OKAP exam.…”

Section: Discussionmentioning

confidence: 99%

“…As the performance of ChatGPT improves (perhaps through prompting strategies), it will be important to work collectively toward building safeguards for our patients 4 . Those will include protecting vulnerable populations from biases and evaluating the potential harm or risk of acting on the answers provided by LLMs like ChatGPT.…”

Section: Discussionmentioning

confidence: 99%

“…DL has been widely used for image recognition using various types of ophthalmic data, such as fundus photographs and optical coherence tomography, and has shown strong results in detecting a wide range of diseases 2 3 . More recently, there has been growing interest in using DL for natural language processing (NLP) in ophthalmology, which involves using AI to understand and interact with human language 4 . NLP has received considerable media attention in the past months due to the release of large DL models called foundation models 5 .…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Antaki

Touma

Milad

et al. 2023

Preprint

150

121

View full text Add to dashboard Cite

We tested the accuracy of ChatGPT, a large language model (LLM), in the ophthalmology question-answering space using two popular multiple choice question banks used for the high-stakes Ophthalmic Knowledge Assessment Program (OKAP) exam. The testing sets were of easy-to-moderate difficulty and were diversified, including recall, interpretation, practical and clinical decision-making problems. ChatGPT achieved 55.8% and 42.7% accuracy in the two 260-question simulated exams. Its performance varied across subspecialties, with the best results in general medicine and the worst in neuro-ophthalmology and ophthalmic pathology and intraocular tumors. These results are encouraging but suggest that specialising LLMs through domain-specific pre-training may be necessary to improve their performance in ophthalmic subspecialties.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Antaki

Touma

Milad

et al. 2023

Preprint

150

121

View full text Add to dashboard Cite

show abstract

“…AI chatbots have recently been developed and refined for multiple fields [ 3 ], including healthcare [ 4 , 5 , 6 , 7 ]. For example, CDS systems that can answer questions have emerged [ 8 , 9 , 10 ].…”

Section: Introductionmentioning

confidence: 99%

“…Therefore, medical research articles related to it are limited. Some GPT-3-related medical articles have been reported about a medical dialogue summarizer [ 28 ], navigation application for electronic health record systems [ 5 ], and its implementation in clinical use [ 7 ] including ophthalmology [ 4 ] and dementia prediction [ 29 ].…”

Section: Introductionmentioning

confidence: 99%

Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study

Hirosawa

Harada

Yokose

et al. 2023

IJERPH

237

114

View full text Add to dashboard Cite

The diagnostic accuracy of differential diagnoses generated by artificial intelligence (AI) chatbots, including the generative pretrained transformer 3 (GPT-3) chatbot (ChatGPT-3) is unknown. This study evaluated the accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical vignettes with common chief complaints. General internal medicine physicians created clinical cases, correct diagnoses, and five differential diagnoses for ten common chief complaints. The rate of correct diagnosis by ChatGPT-3 within the ten differential-diagnosis lists was 28/30 (93.3%). The rate of correct diagnosis by physicians was still superior to that by ChatGPT-3 within the five differential-diagnosis lists (98.3% vs. 83.3%, p = 0.03). The rate of correct diagnosis by physicians was also superior to that by ChatGPT-3 in the top diagnosis (53.3% vs. 93.3%, p < 0.001). The rate of consistent differential diagnoses among physicians within the ten differential-diagnosis lists generated by ChatGPT-3 was 62/88 (70.5%). In summary, this study demonstrates the high diagnostic accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical cases with common chief complaints. This suggests that AI chatbots such as ChatGPT-3 can generate a well-differentiated diagnosis list for common chief complaints. However, the order of these lists can be improved in the future.

show abstract

Advancing systemic disease diagnosis through ophthalmic image‐based artificial intelligence

Miao,

Zou,

et al. 2024

MedComm – Future Medicine

View full text Add to dashboard Cite

The eye serves as a unique window into systemic health, offering clinicians a valuable opportunity for early detection and targeted treatment. Against this backdrop, advancements in artificial intelligence (AI) and ophthalmic imaging are converging to pave the way for more precise and predictive diagnostics. This review aims to elucidate the transformative role of AI in utilizing ophthalmic imaging for the detection and prediction of systemic diseases. We begin by introducing the advantages of the eye as a valuable tool for detecting systemic diseases. We also provide an overview of various ophthalmic imaging techniques that have proven useful in predicting systemic ailments. Then, we summarize two research patterns for analyzing ocular data, followed by the introduction of current AI applications using ophthalmic images that significantly increase diagnostic precision. Despite the promise, challenges such as data heterogeneity and model interpretability persist, which are also covered in this review. We conclude by discussing future directions and the immense potential these AI‐enabled approaches hold for revolutionizing healthcare. As AI technologies advance, their potential integration with ophthalmic imaging offers promising avenues for improving the diagnosis, prediction, and management of various systemic diseases, thereby contributing to the evolving landscape of integrated healthcare.

show abstract

New meaning for NLP: the trials and tribulations of natural language processing with GPT-3 in ophthalmology

Cited by 90 publications

References 25 publications

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study

Advancing systemic disease diagnosis through ophthalmic image‐based artificial intelligence

Contact Info

Product

Resources

About