Corolla: An Efficient Multi-Modality Fusion Framework with Supervised Contrastive Learning for Glaucoma Grading

Cai, Zhiyuan; Lin, Li; He, Huaqing; Tang, Xiaoying

doi:10.1109/isbi52829.2022.9761712

Cited by 13 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, some works introduce contrastive learning into supervised tasks by providing supervised signals through labels rather than pretext tasks. These works can be categorized into object-wise contrast [36], [37] and pixel-wise contrast [38]- [40]. For the former, images are treated as samples and tasks are generally classification.…”

Section: Related Workmentioning

confidence: 99%

Feature-Fusion Segmentation Network for Landslide Detection Using High-Resolution Remote Sensing Images and Digital Elevation Model Data

Liu

Peng

et al. 2023

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

As a harzard disaster, landslide often brings tremendous losses to humanity, so it's necessary to achieve reliable detection of landslide. However, the problems of visual blur and small-sized dataset cause great challenges for old landslide detection task when using remote sensing data. To reliably extract semantic features, a hyper-pixel-wise contrastive learning augmented segmentation network (HPCL-Net) is proposed, which augments the local salient feature extraction from the boundaries of landslides through HPCL and fuses the heterogeneous infromation in the semantic space from High-Resolution Remote Sensing Images and Digital Elevation Model Data data. For full utilization of the precious samples, a global hyper-pixel-wise sample pair queues-based contrastive learning method, which includes the construction of global queues that store hyperpixel-wise samples and the updating scheme of a momentum encoder, is developed, reliably enhancing the extraction ability of semantic features. The proposed HPCL-Net is evaluated on a Loess Plateau old landslide dataset and experiment results show that the model greatly improves the reliablity of old landslide detection compared to the previous old landslide segmentation model, where mIoU metric is increased from 0.620 to 0.651, Landslide IoU metric is increased from 0.334 to 0.394 and F1score metric is increased from 0.501 to 0.565.

show abstract

Section: Related Workmentioning

confidence: 99%

Feature-Fusion Segmentation Network for Landslide Detection Using High-Resolution Remote Sensing Images and Digital Elevation Model Data

Liu

Peng

et al. 2023

IEEE Trans. Geosci. Remote Sensing

View full text Add to dashboard Cite

show abstract

“…In addition, Huang et al [40] proposed a self-supervised model for lesion-based contrastive learning for DR grading, and it performs well on the EyePACS dataset. Cai et al [41] used optical coherence tomography (OCT) and CFP for multi-modality supervised contrastive learning to diagnose glaucoma. In two papers, Cheng et al [42,43] used two contrastive learning methods to improve the imaging quality of fundus images.…”

Section: Related Workmentioning

confidence: 99%

Supervised Contrastive Learning with Angular Margin for the Detection and Grading of Diabetic Retinopathy

Zhu

Chen

et al. 2023

Diagnostics

View full text Add to dashboard Cite

Many researchers have realized the intelligent medical diagnosis of diabetic retinopathy (DR) from fundus images by using deep learning methods, including supervised contrastive learning (SupCon). However, although SupCon brings label information into the calculation of contrastive learning, it does not distinguish between augmented positives and same-label positives. As a result, we propose the concept of Angular Margin and incorporate it into SupCon to address this issue. To demonstrate the effectiveness of our strategy, we tested it on two datasets for the detection and grading of DR. To align with previous work, Accuracy, Precision, Recall, F1, and AUC were selected as evaluation metrics. Moreover, we also chose alignment and uniformity to verify the effect of representation learning and UMAP (Uniform Manifold Approximation and Projection) to visualize fundus image embeddings. In summary, DR detection achieved state-of-the-art results across all metrics, with Accuracy = 98.91, Precision = 98.93, Recall = 98.90, F1 = 98.91, and AUC = 99.80. The grading also attained state-of-the-art results in terms of Accuracy and AUC, which were 85.61 and 93.97, respectively. The experimental results demonstrate that Angular Margin is an excellent intelligent medical diagnostic algorithm, performing well in both DR detection and grading tasks.

show abstract

“…At the same time, transformers have also made rapid progress in the field of computer vision (Dosovitskiy et al 2020, Liu et al 2021, Touvron et al 2021, and transformers have demonstrated their unique advantages in the field of computer vision. Currently, some studies use both modalities and fuse the features of the two modalities (Cai et al 2022, Wu et al 2022. Compared to single modality, multimodal is often a hot research topic in medical imaging, and it can often achieve better results (Zhou et al 2019).…”

Section: Introduction: File Preparation and Submissionmentioning

confidence: 99%

Mstnet: method for glaucoma grading based on multimodal feature fusion of spatial relations

Wang,

Zhang

et al. 2023

Phys. Med. Biol.

View full text Add to dashboard Cite

Objective:The objective of this study is to develop an efficient multimodal learning framework for the classification of glaucoma. Glaucoma is a group of eye diseases that can result in vision loss and blindness, often due to delayed detection and treatment. Fundus images and optical coherence tomography (OCT) images have proven valuable for the diagnosis and management of glaucoma. However, current models that combine features from both modalities often lack efficient spatial relationship modeling. Approach:In this study, we propose an innovative approach to address the classification of glaucoma. We focus on leveraging the features of OCT volumes and harness the capabilities of transformer models to capture long-range spatial relationships. To achieve this, we introduce a 3D transformer model to extract features from OCT volumes, enhancing the model's effectiveness. Additionally, we employ downsampling techniques to enhance model efficiency. We then utilize the spatial feature relationships between OCT volumes and fundus images to fuse the features extracted from both sources. Main Results:Our proposed framework has yielded remarkable results, particularly in terms of glaucoma grading performance. We conducted our experiments using the GAMMA dataset, and our approach outperformed traditional feature fusion methods. By effectively modeling spatial relationships and combining OCT volume and fundus map features, our framework achieved outstanding classification results.Significance:This research is of significant importance in the field of glaucoma diagnosis and management. Efficient and accurate glaucoma classification is essential for timely intervention and prevention of vision loss. Our proposed approach, which integrates 3D transformer models, offers a novel way to extract and fuse features from OCT volumes and fundus images, ultimately enhancing the effectiveness of glaucoma classification. This work has the potential to contribute to improved patient care, particularly in the early detection and treatment of glaucoma, thereby reducing the risk of vision impairment and blindness.

show abstract

Corolla: An Efficient Multi-Modality Fusion Framework with Supervised Contrastive Learning for Glaucoma Grading

Cited by 13 publications

References 11 publications

Feature-Fusion Segmentation Network for Landslide Detection Using High-Resolution Remote Sensing Images and Digital Elevation Model Data

Feature-Fusion Segmentation Network for Landslide Detection Using High-Resolution Remote Sensing Images and Digital Elevation Model Data

Supervised Contrastive Learning with Angular Margin for the Detection and Grading of Diabetic Retinopathy

Mstnet: method for glaucoma grading based on multimodal feature fusion of spatial relations

Contact Info

Product

Resources

About