Mask Based Unsupervised Content Transfer

Abstract

We consider the problem of translating, in an unsupervised manner, between two domains where one contains some additional information compared to the other. The proposed method disentangles the common and separate parts of these domains and, through the generation of a mask, focuses the attention of the underlying network to the desired augmentation alone, without wastefully reconstructing the entire target. This enables state-of-the-art quality and variety of content translation, as shown through extensive quantitative and qualitative evaluation. Furthermore, the novel mask-based formulation and regularization is accurate enough to achieve state-of-the-art performance in the realm of weakly supervised segmentation, where only class labels are given. To our knowledge, this is the first report that bridges the problems of domain disentanglement and weakly supervised segmentation. Our code is publicly available at https://github.com/rmokady/mbu-content-tansfer.Preprint. Under review.

JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting

“…UNIT [22] and MUNIT [14], for example, use a shared representation for image-to-image translation. Other works [29,6,25] use shared representations to disentangle the common content of two domains from the separate part. Unlike these methods, our work disentangles motion from style over videos.…”

“…The generator consists of 9 residual blocks, each contains convolution, ReLU, and Instance Normalization layers. The discriminator consists of 3 fully connected and Leaky ReLU layers, followed by a final sigmoid activation, similarly to Mokady et al [25].…”