MulTiNet: Multimodal Neural Networks for Glaucoma Based on Transfer Learning

Abstract

Background: Being one of the most serious causes of irreversible blindness, glaucoma has many subtypes and complex symptoms. In clinic, doctors usually need to use a variety of medical images for diagnosis. Optical Coherence Tomography (OCT), Visual Field (VF) , Fundus Photosexams (FP) and Ultrasonic BioMicroscope (UBM) are widely-used and complementary techniques for diagnosing glaucoma.Methods: At present, the field of intelligent diagnosis of glaucoma is limited by two major problems. One is the small number of data sets, and the other is the low diagnostic accuracy of Single-Modal Modal. In order to solve the above two problems, we have done the following work. First, we construct DualSY glaucoma multimodal data set. The four most important subtypes of glaucoma are discussed in this article which are Primary Open Angle Glaucoma (POAG), Primary Angle Closure Glaucoma (PACG), Primary Angle Closure Suspect (PACS) and Primary Angle Closure (PAC). Each patient in the DualSY data set contains more than five medical images, as shown in the figure 4.And DualSY are labeled with image-level multi-labels. Second, We propose a new Multi-Modal classification network for glaucoma, which is a multiclass classification model with various medical images of glaucoma patients and text information as input. The network structure consists of three main branches to deal with patient metadata, domain-based glaucoma features and medical images. Transfer learning method is introduced into this paper due to the small number of medical image data sets. The flowchart is shown in Figure 5.Result: Our method on glaucoma diagnosis outperforms state-of-the-art methods. A promising average result of overall accuracy (ACC) of 94.7% is obtained. Our data set outperformed most data sets in glaucoma diagnosis with an accuracy of 87.8%.Conclusions: The results suggest that medical images such as Heidelberg OCT and three-dimensional fundus photos used in this paper can better express the high-level information of glaucoma and our modal greatly improve the accuracy of glaucoma diagnosis. At the same time, this data set has great potential, and we continue to study this data.

An effective deep learning network for detecting and classifying glaucomatous eye

“…However, feature extraction was also utilized with CDR and brought 92% accuracy. Li et al [16] presented a unique approach that can detect glaucomatous eyes. After collecting the data, they applied several deep learning networks and got 87.8% accuracy.…”