Generative Adversarial Networks for Video-to-Video Domain Adaptation

Abstract

Endoscopic videos from multicentres often have different imaging conditions, e.g., color and illumination, which make the models trained on one domain usually fail to generalize well to another. Domain adaptation is one of the potential solutions to address the problem. However, few of existing works focused on the translation of video-based data. In this work, we propose a novel generative adversarial network (GAN), namely VideoGAN, to transfer the video-based data across different domains. As the frames of a video may have similar content and imaging conditions, the proposed VideoGAN has an X-shape generator to preserve the intra-video consistency during translation. Furthermore, a loss function, namely color histogram loss, is proposed to tune the color distribution of each translated frame. Two colonoscopic datasets from different centres, i.e., CVC-Clinic and ETIS-Larib, are adopted to evaluate the performance of domain adaptation of our VideoGAN. Experimental results demonstrate that the adapted colonoscopic video generated by our VideoGAN can significantly boost the segmentation accuracy, i.e., an improvement of 5%, of colorectal polyps on multicentre datasets. As our VideoGAN is a general network architecture, we also evaluate its performance with the CamVid driving video dataset on the cloudy-to-sunny translation task. Comprehensive experiments show that the domain gap could be substantially narrowed down by our VideoGAN.

Mutual-GAN: Towards Unsupervised Cross-Weather Adaptation with Mutual Information Constraint

“…applications of the existing frameworks to the domain adaptation of urban scenes, which requires rigorous preservation of image-objects. The VideoGAN [5] can maintain the image-objects, but fail to translate the image-weather. The clouds are clearly observed in its cloudy-to-sunny translated results.…”

“…To reduce the workload of manual annotation, we try to address the problem of content distortion in an unsupervised manner. Chen et al [5] implemented a VideoGAN to maintain the style-consistency across a driving video during unsupervised video-to-video translation. Xie et al [28] proposed an unsupervised cross-weather adaptation approach, namely OP-GAN, by integrating a self-supervised module into the CycleGAN.…”

“…Several unpaired image-to-image domain adaptation frameworks, including CycleGAN [33], UNIT [20] and DRIT [18] are taken as baselines for the performance evaluation. The state-of-the-art frameworks (i.e., VideoGAN [5] and OP-GAN [28]) and the direct transfer approach, which directly takes the source domain data for testing without any adaptation, are also involved for comparison. 4 Note that the recently proposed GAN for imagebased domain adaptation, i.e., AugGAN [15], is not involved for comparison, due to the strong prior-knowledge (pixel-wise annotation) used in the approach.…”

“…The benchmarking algorithms adopted in this study is the same to the previous studies [15,5,28]. Some of the existing unsupervised domain adaptation (UDA) methods [19,4,30,27,35] are not involved for comparison, since they are feature-alignment (FA) based, while our Mutual-GAN is an image-to-image-based domain adaptation method.…”

SUMMER: Bias-aware Prediction of Graduate Employment Based on Educational Big Data

“…The main idea here is that we use the GAN (generative adversarial networks)-related model to learn the distribution behind the data and then generate new data samples to balance the dataset. GAN is a generative model based on neural network structure, which is widely used in various ields [8,55]. The GAN training strategy is a game between two competing networks: the generator and the discriminator.…”