Colonoscopy Polyp Detection: Domain Adaptation From Medical Report Images to Real-time Videos

Abstract

Automatic colorectal polyp detection in colonoscopy video is a fundamental task, which has received a lot of attention. Manually annotating polyp region in a large scale video dataset is time-consuming and expensive, which limits the development of deep learning techniques. A compromise is to train the target model by using labeled images and infer on colonoscopy videos. However, there are several issues between the image-based training and video-based inference, including domain differences, lack of positive samples, and temporal smoothness. To address these issues, we propose an Image-video-joint polyp detection network (Ivy-Net) to address the domain gap between colonoscopy images from historical medical reports and real-time videos. In our Ivy-Net, a modified mixup is utilized to generate training data by combining the positive images and negative video frames at the pixel level, which could learn the domain adaptive representations and augment the positive samples. Simultaneously, a temporal coherence regularization (TCR) is proposed to introduce the smooth constraint on feature-level in adjacent frames and improve polyp detection by unlabeled colonoscopy videos. For evaluation, a new large colonoscopy polyp dataset is collected, which contains 3056 images from historical medical reports of 889 positive patients and 7.5-hour videos of 69 patients (28 positive). The experiments on the collected dataset demonstrate that our Ivy-Net achieves the state-of-the-art result on colonoscopy video.

The Advent of Domain Adaptation into Artificial Intelligence for Gastrointestinal Endoscopy and Medical Imaging

“…In Ivy-Net, a modified mix-up is utilized to generate training data by combining positive and negative video frames at the pixel level, which can learn domain-adaptive representations and augment positive samples. Experiments on collected datasets have demonstrated that Ivy-Net achieves state-of-the-art results and significantly improves the average precision of the polyp detection in colonoscopy videos [26][27][28]. It is not surprising that, if a model is trained with data from an older capsule, it may not yield the expected results when it is evaluated with a newer capsule, because the same distribution of data is not guaranteed.…”

“…Laiz et al first proposed the concept of triplet loss function for domain adaptation in capsule endoscopy, and reported that it could improve the reading accuracy by improving the generalization of the datasets obtained from different systems [25]. Zhan et al improved the polyp detection in colonoscopy videos by using Ivy-Net to bridge the domain gap between colonoscopy images and real-time videos [28]. Celik et al applied a UDA framework for the segmentation of BE, which is a precancerous lesion [32].…”