Recommendations for initial diabetic retinopathy screening of diabetic patients using large language model-based artificial intelligence in real-life case scenarios

Abstract

Purpose To study the role of artificial intelligence (AI) to identify key risk factors for diabetic retinopathy (DR) screening and develop recommendations based on clinician and large language model (LLM) based AI platform opinions for newly detected diabetes mellitus (DM) cases. Methods Five clinicians and three AI applications were given 20 AI-generated hypothetical case scenarios to assess DR screening timing. We calculated inter-rater agreements between clinicians, AI-platforms, and the “majority clinician response” (defined as the maximum number of identical responses provided by the clinicians) and “majority AI-platform” (defined as the maximum number of identical responses among the 3 distinct AI). Scoring was used to identify risk factors of different severity. Three, two, and one points were given to risk factors requiring screening immediately, within a year, and within five years, respectively. After calculating a cumulative screening score, categories were assigned. Results Clinicians, AI platforms, and the “majority clinician response” and “majority AI response” had fair inter-rater reliability (k value: 0.21–0.40). Uncontrolled DM and systemic co-morbidities required immediate screening, while family history of DM and a co-existing pregnancy required screening within a year. The absence of these risk factors required screening within 5 years of DM diagnosis. Screening scores in this study were between 0 and 10. Cases with screening scores of 0–2 needed screening within 5 years, 3–5 within 1 year, and 6–12 immediately. Conclusion Based on the findings of this study, AI could play a critical role in DR screening of newly diagnosed DM patients by developing a novel DR screening score. Future studies would be required to validate the DR screening score before it could be used as a reference in real-life clinical situations. Clinical trial registration Not applicable.

Artificial Intelligence Applications in Diabetic Retinopathy: What We Have Now and What to Expect in the Future

“…From simple consultations and facilitating appointment scheduling to providing advice on diseases, LLMs are expected to serve as decision-making models between doctors and patients [ 59 ]. Gopalakrishnan et al [ 60 ] explored the application of LLMs in the domain of DR. They conducted an analysis of DR screening timing through 20 hypothetical case scenarios using inputs from five clinicians and three AI applications.…”

“…They conducted an analysis of DR screening timing through 20 hypothetical case scenarios using inputs from five clinicians and three AI applications. They reported fair inter-rater reliability (κ=0.21–0.4) between “major clinician response” and “major AI response” [ 60 ]. Such endeavors are deemed to be highly beneficial tools for patients who have difficulty accessing ophthalmologists, and they provide insight into the potential evolution of LLMs in the field of DR.…”