Crowdsourcing for Artificial Intelligence Models in Ophthalmology

Abstract

In this issue of JAMA Ophthalmology, Qian et al 1 present their research on artificial intelligence (AI) and deep learning algorithms for diagnosing myopic maculopathy (MM). By organizing an international competition called the Myopic Maculopathy Analysis Challenge (MMAC), 2 the authors aimed to develop and evaluate automated solutions for MM classification, lesion segmentation, and spherical equivalent prediction. This competition framework provided a standardized approach for creating and assessing AI models compared with the diagnostic performance of ophthalmologists.The MMAC competition distributed 3 subdatasets containing a total of 2306, 294, and 2003 fundus images to participants. These images, captured using color fundus photography, were provided to develop algorithms that were subsequently evaluated on independent test sets. The competition's structure facilitated a rigorous comparison of AI models, with results showing that the best-performing algorithm achieved a very good quadratic-weighted κ of 0.901 for MM classification and a dice similarity coefficient of up to 0.841 for Fuchs spot segmentation. Model ensembles, combining outputs from multiple algorithms, showed enhanced performance in MM classification and segmentation tasks, surpassing individual models and ophthalmologists.This approach, known as crowdsourcing science, leverages public capacity to address research questions. Since its introduction by Howe in 2006, 3 crowdsourcing has gained significant attention in the literature, becoming a priority for the National Library of Medicine and the National Institutes of Health (NIH). Recently, the NIH has funded 4 datageneration projects totaling $130 million to accelerate AI usage in biomedical research aimed at solving real-world problems. These projects generate flagship datasets, tools, and practice guides representing best practices in using AI in the biomedical sector 4 and are designed for broader use by the research community. Widespread sharing of these datasets and others supported by the NIH Common Fund are aimed at fueling advancements by lowering barriers to data access. Similarly, the California Healthcare Foundation sponsored a Kaggle competition in 2015 to develop models for detecting diabetic retinopathy (DR) severity. 5 In that competition, the ground truth was determined by a single clinician. In contrast, in the work by Qian et al, 1 3 specialists with different experience levels labeled the fundus photographs, highlighting a more rigorous approach to ground truth definition. This method has empowered the modeling efforts by incorporating input from experienced experts. Notably, the ensemble approach achieved higher accuracy in segmentation tasks, which ophthalmologists are not trained to perform. On the classification front, the ensemble model surpassed ophthalmologists by 7.4% higher sensitivity and 1.3% higher specificity in diagnosing MM.