FFA-GPT: an automated pipeline for fundus fluorescein angiography interpretation and question-answer

Abstract

Fundus fluorescein angiography (FFA) is a crucial diagnostic tool for chorioretinal diseases, but its interpretation requires significant expertise and time. Prior studies have used Artificial Intelligence (AI)-based systems to assist FFA interpretation, but these systems lack user interaction and comprehensive evaluation by ophthalmologists. Here, we used large language models (LLMs) to develop an automated interpretation pipeline for both report generation and medical question-answering (QA) for FFA images. The pipeline comprises two parts: an image-text alignment module (Bootstrapping Language-Image Pre-training) for report generation and an LLM (Llama 2) for interactive QA. The model was developed using 654,343 FFA images with 9392 reports. It was evaluated both automatically, using language-based and classification-based metrics, and manually by three experienced ophthalmologists. The automatic evaluation of the generated reports demonstrated that the system can generate coherent and comprehensible free-text reports, achieving a BERTScore of 0.70 and F1 scores ranging from 0.64 to 0.82 for detecting top-5 retinal conditions. The manual evaluation revealed acceptable accuracy (68.3%, Kappa 0.746) and completeness (62.3%, Kappa 0.739) of the generated reports. The generated free-form answers were evaluated manually, with the majority meeting the ophthalmologists’ criteria (error-free: 70.7%, complete: 84.0%, harmless: 93.7%, satisfied: 65.3%, Kappa: 0.762–0.834). This study introduces an innovative framework that combines multi-modal transformers and LLMs, enhancing ophthalmic image interpretation, and facilitating interactive communications during medical consultation.

ChatFFA: An ophthalmic chat system for unified vision-language understanding and question answering for fundus fluorescein angiography

“… 18 These models can be directly integrated into the cross-modal model to overcome the limitations in model output and provide further interactive explanations. 19 , 20 Additionally, they can serve as external sources of knowledge and semantic refiners to optimize the fine-tuning process of cross-modal models. This enables the models to accurately capture details and semantic information in images, facilitating generalization to unseen objects or concepts and performing more complex downstream tasks.…”

“…FFA images were retrospectively collected from a tertiary center between 2016 and 2019. 20 After data cleaning by removing duplicates, handling missing values, and correcting erroneous data, a total of 654,343 FFA images from 9,392 patients were used for model development. All patient data were anonymized and de-identified, as well as stored in a secure data center, managed and analyzed by authorized personnel.…”

Unveiling the Clinical Incapabilities: A Benchmarking Study of GPT-4V(ision) for Ophthalmic Multimodal Image Analysis

“…[24] Another way to improve the model is to align the ophthalmic image features with medical texts. [11 25] Additionally, researchers may improve their models by applying transfer learning, fine-tuning, and reinforcement learning techniques, or by incorporating knowledge enrichment methods like Retrieval-Augmented Generation (RAG). [26 27] However, before applying any model to clinical practice, detailed clinical trials should be conducted to assess its safety and efficacy.…”

“…[10] However, ophthalmology was not included in the study, leaving the practical application capabilities of GPT-4V in addressing image-related concerns in ophthalmology uncertain. Meanwhile, previous work on ophthalmic VQA focuses on a specific modality,[11 12] leaving multimodal ocular VQA unexplored.…”