Image-To-Image Translation via Group-Wise Deep Whitening-And-Coloring Transformation

Abstract

Recently, unsupervised exemplar-based image-to-image translation, conditioned on a given exemplar without the paired data, has accomplished substantial advancements.In order to transfer the information from an exemplar to an input image, existing methods often use a normalization technique, e.g., adaptive instance normalization, that controls the channel-wise statistics of an input activation map at a particular layer, such as the mean and the variance. Meanwhile, style transfer approaches similar task to image translation by nature, demonstrated superior performance by using the higher-order statistics such as covariance among channels in representing a style. In detail, it works via whitening (given a zero-mean input feature, transforming its covariance matrix into the identity). followed by coloring (changing the covariance matrix of the whitened feature to those of the style feature). However, applying this approach in image translation is computationally intensive and error-prone due to the expensive time complexity and its non-trivial backpropagation. In response, this paper proposes an end-to-end approach tailored for image translation that efficiently approximates this transformation with our novel regularization methods. We further extend our approach to a group-wise form for memory and time efficiency as well as image quality. Extensive qualitative and quantitative experiments demonstrate that our proposed method is fast, both in training and inference, and highly effective in reflecting the style of an exemplar. Finally, our code is available at https://github.com/ WonwoongCho/GDWCT.

UVIRT—Unsupervised Virtual Try-on Using Disentangled Clothing and Person Features

“…Next, although we want to reflect the style features to the content features, inspired by MUNIT, the content and style features highly are entangled when using Adaptive Instance Normalization (AdaIN) [ 33 ]. Hence, we apply the style features to the exchanged content features by using Group-wise Deep Whitening-and-Coloring Transformation (GDWCT) [ 30 ] in order to not entangle the content and style features. Briefly, after the content feature gets whitened, using the mean of the content features (Group-wise Deep Whitening Translation: GDWT), the style feature is applied to the whitened content feature (Group-wise Deep Coloring Translation: GDCT).…”

“…Our model loss function is the same as in the GDWCT paper [ 30 ]. We describe the ten loss functions and two regularization terms in the following text.…”

“…The two encoders which extract the style features have a fully convolutional network, which consists of one convolutional layer, four downsample convolutional layers, and a global average pooling layer, sequentially. The two decoders are structured as a ResNet-like network, which consists of the GDWCT module whose architecture is the same as in Reference [ 30 ]. The two discriminators have fully convolutional networks, which consist of three downsample convolutional layers and one point-wise convolutional layer, sequentially.…”

“…In our experiments, although we first employed MUNIT, it could not warp clothing to people well. Therefore, we utilize the state-of-the-art I2I assistant method “Group-wise Deep Whitening-and-Coloring Translation (GDWCT)” [ 30 ]. GDWCT can warp a clothing feature to a person as it can transfer domain-specific features well.…”

Fighting Deepfake by Exposing the Convolutional Traces on Images

“…Many methods work in the latent space representation in order to set constraints to the attributes to be modified, an example is ATTGAN, created by He et al [11]. Cho et al [12] proposed the "groupwise deep whitening-and coloring method" (GDWCT) for a better styling capacity, obtaining a great improvement in the image translation and style transfer task in terms of computational efficiency and quality of generated images. The stage changes when surprising results of Deepfake images were obtained by Style Generative Adversarial Network (STYLEGAN) [13].…”