Recycle-GAN: Unsupervised Video Retargeting

Abstract

We introduce a data-driven approach for unsupervised video retargeting that translates content from one domain to another while preserving the style native to a domain, i.e., if contents of John Oliver's speech were to be transferred to Stephen Colbert, then the generated content/speech should be in Stephen Colbert's style. Our approach combines both spatial and temporal information along with adversarial losses for content translation and style preservation. In this work, we first study the advantages of using spatiotemporal constraints over spatial constraints for effective retargeting. We then demonstrate the proposed approach for the problems where information in both space and time matters such as face-to-face translation, flower-to-flower, wind and cloud synthesis, sunrise and sunset.

Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation

“…To the best of our knowledge, our approach is the first use-case of optical flow in an unsupervised video translation setting. Our experiments show that the warping enables the re-use of the pixels from the previous output, and this yields better video results than the RecycleGAN [1] method which directly predicts all pixels / frames from scratch.…”

“…The original contents (i.e.semantic labels) in a video tend to be mistranslated by existing CycleGAN-based frameworks [1,41]. Indeed, conventional cycle-consistency does not necessarily guarantee the translation to be semantically consistent.…”

“…CycleGAN [41] 25 Warping error. CycleGAN [41] 0.002017 CycleGAN [41]+BT [17] 0.000829 CyCADA [10] 0.002003 CyCADA [10]+BT [17] 0.000945 RecycleGAN [1] 0.001640 Ours 0.000437 Oracle -Source (VIPER) 0.000264 Table 3: Ablation studies. We compute the semantic segmentation score and the flow warping loss to quantitatively determine the effects of the proposed loss functions.…”

“…Method road sidewalk building wall fence pole traffic light traffic sign vegetation terrain sky person rider car truck bus train motorbike bicycle Table 4: Domain adaptation results. We compared the proposed method to CycleGAN [41], CyCADA [10], and RecycleGAN [1]. We also experimented using only the source, as well as having data from the target domain.…”

Temporarily-Aware Context Modeling Using Generative Adversarial Networks for Speech Activity Detection

“…The objective in Eq. 6 is shown to be highly effective for generating realistic static outputs considering the elements of the sequence individually [49]. However, it discards the temporal coherence as the generator and the discriminator consider each frame individually [49].…”