Joint Vessel Segmentation and Deformable Registration on Multi-Modal Retinal Images Based on Style Transfer

Zhang, Junkang; An, Cheolhong; Dai, Ji; Amador, Manuel; Bartsch, Dirk-Uwe; Borooah, Shyamanga; Freeman, William R.; Nguyen, Truong Q.

doi:10.1109/icip.2019.8802932

Cited by 21 publications

(31 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our AI overlay strategy consisted of a joint vessel segmentation and a deformable registration model based on the convolutional neural network because retina vessels are key landmarks even for different imaging modalities. 18 , 19 The proposed learning scheme utilized two learning networks. First, a style transfer network was applied 20 to train a vessel segmentation without ground truth such that it would extract mutual patterns between multimodal retinal images to find good correspondences.…”

Section: Methodsmentioning

confidence: 99%

“…1 ). 18 The style transfer network transformed input retina images to target style images (segmented vessel images) as shown in Figure 1 . The style transfer network (STN) uses a pre-trained convolutional neural network (CNN) to model the global vessel structure with an outside dataset (represented by the image in Figure 1 ).…”

Section: Methodsmentioning

confidence: 99%

“… 23 The registration network provides the alignment information from a source image (CF image) to a target image (SLO image and aligned both segmented vessel images of two modalities and original retina images. The Registration Network consists of the super-point network 24 for key point detection and description, the outlier rejection network 25 for reliable matching points selection, and the refinement network 18 for sub-pixel level adjustment. The registration network was also trained without any labeled data because it is impossible to obtain dense correspondences for retina images.…”

Section: Methodsmentioning

confidence: 99%

“…More details on the network or reports to replicate the results have previously been published for us. 18 , 19 We used 109 datasets to train the algorithm, 20 datasets for validation and 89 datasets for testing.…”

Section: Methodsmentioning

confidence: 99%

“… Diagram of the style transfer network (STN). 18 ( A ) Shows the pretraining of the CNN using a single pair of fundus image and segmented vessel diagram. This image was not part of our dataset and was used to train our STN to detect tree-like structures, continuously stretching and branching vessel paths with decreasing width, etc.…”

Section: Methodsmentioning

confidence: 99%

See 4 more Smart Citations

Artificial Intelligence for Automated Overlay of Fundus Camera and Scanning Laser Ophthalmoscope Images

Cavichini

Bartsch

et al. 2020

Trans. Vis. Sci. Tech.

Self Cite

View full text Add to dashboard Cite

Purpose The purpose of this study was to evaluate the ability to align two types of retinal images taken on different platforms; color fundus (CF) photographs and infrared scanning laser ophthalmoscope (IR SLO) images using mathematical warping and artificial intelligence (AI). Methods We collected 109 matched pairs of CF and IR SLO images. An AI algorithm utilizing two separate networks was developed. A style transfer network (STN) was used to segment vessel structures. A registration network was used to align the segmented images to each. Neither network used a ground truth dataset. A conventional image warping algorithm was used as a control. Software displayed image pairs as a 5 × 5 checkerboard grid composed of alternating subimages. This technique permitted vessel alignment determination by human observers and 5 masked graders evaluated alignment by the AI and conventional warping in 25 fields for each image. Results Our new AI method was superior to conventional warping at generating vessel alignment as judged by masked human graders ( P < 0.0001). The average number of good/excellent matches increased from 90.5% to 94.4% with AI method. Conclusions AI permitted a more accurate overlay of CF and IR SLO images than conventional mathematical warping. This is a first step toward developing an AI that could allow overlay of all types of fundus images by utilizing vascular landmarks. Translational Relevance The ability to align and overlay imaging data from multiple instruments and manufacturers will permit better analysis of this complex data helping understand disease and predict treatment.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%