Unsupervised Image-to-Image Translation with Generative Prior

Yang, Shuai; Jiang, Liming; Liu, Ziwei; Loy, Chen Change

doi:10.1109/cvpr52688.2022.01779

Cited by 39 publications

(15 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MMFS achieves both good stylization and faithful facial detail preserving. DualStyleGAN [YJLL22a] requires an input exemplar as the reference and we randomly select four images from the AAHQ dataset as exemplars for stylization.…”

Section: Methodsmentioning

confidence: 99%

Multi‐Modal Face Stylization with a Generative Prior

Li,

Dong,

Lin

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

In this work, we introduce a new approach for face stylization. Despite existing methods achieving impressive results in this task, there is still room for improvement in generating high‐quality artistic faces with diverse styles and accurate facial reconstruction. Our proposed framework, MMFS, supports multi‐modal face stylization by leveraging the strengths of StyleGAN and integrates it into an encoder‐decoder architecture. Specifically, we use the mid‐resolution and high‐resolution layers of StyleGAN as the decoder to generate high‐quality faces, while aligning its low‐resolution layer with the encoder to extract and preserve input facial details. We also introduce a two‐stage training strategy, where we train the encoder in the first stage to align the feature maps with StyleGAN and enable a faithful reconstruction of input faces. In the second stage, the entire network is fine‐tuned with artistic data for stylized face generation. To enable the fine‐tuned model to be applied in zero‐shot and one‐shot stylization tasks, we train an additional mapping network from the large‐scale Contrastive‐Language‐Image‐Pre‐training (CLIP) space to a latent w+ space of fine‐tuned StyleGAN. Qualitative and quantitative experiments show that our framework achieves superior performance in both one‐shot and zero‐shot face stylization tasks, outperforming state‐of‐the‐art methods by a large margin.

show abstract

Section: Methodsmentioning

confidence: 99%

Multi‐Modal Face Stylization with a Generative Prior

Li,

Dong,

Lin

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…Dynamic PDGAN introduces the pretrained teacher discriminator to boost the student StyleGAN with an uncompressed architecture. Most previous works 25 – 29 focusing on the distillation of GANs are designed for the image-to-image translation task, aligning the student generator with the teacher generator in the output/feature space. A few previous works 40 , 41 are designed for the noise-to-image task, aiming at boosting the performance of compressed GANs from the perspective of knowledge distillation.…”

Section: Related Workmentioning

confidence: 99%

“…Thus, it is sub-optimal to align features or outputs of the teacher generator and the student generator like in image-to-image translation tasks 25 – 29 The manner to utilize the knowledge contained in the teacher model needs to be thoroughly explored. Second, it is highly non-trivial to obtain a student model that can surpass the teacher model, indicating that only transferring the knowledge of the teacher model cannot guarantee a superior student model.…”

Section: Introductionmentioning

confidence: 99%

Dynamic PDGAN: discriminator-boosted knowledge distillation for StyleGANs

Tian,

Shen,

Tian

et al. 2024

J. Electron. Imag.

View full text Add to dashboard Cite

Generative adversarial networks have shown remarkable success in image synthesis, especially StyleGANs. Equipped with delicate and specific designs, StyleGANs are capable of synthesizing high-resolution and high-fidelity images. Previous works aiming at improving StyleGANs mainly focus on modifying the architecture of StyleGANs or transferring knowledge from other domains. However, the knowledge contained in StyleGANs trained in the same domain is still unexplored. We aim to further boost the performance of StyleGANs from the perspective of knowledge distillation, i.e., improving uncompressed StyleGANs with the aid of teacher StyleGANs trained in the same domain. Motivated by the implicit distribution contained in the pretrained teacher discriminator, we propose to exploit the teacher discriminator to additionally supervise the student generator of StyleGANs so as to leverage the knowledge in the teacher discriminator. With the proposed distillation scheme, our method can outperform original StyleGANs on several large-scale datasets, achieving state-of-the-art on AFHQv2.

show abstract

“…Besides, the encoding structure of GANs makes it difficult to decouple appearance and structural information. When the gap between the two domains is too large, the result may not be transformed [23]- [25] or have lost information from the original domain [26].…”

Section: B Gan-based Image Transfermentioning

confidence: 99%

DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Models

Cao¹,

Hao²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Image-based fashion design with AI techniques has attracted increasing attention in recent years. We focus on a new fashion design task, where we aim to transfer a reference appearance image onto a clothing image while preserving the structure of the clothing image. It is a challenging task since there are no reference images available for the newly designed output fashion images. Although diffusion-based image translation or neural style transfer (NST) has enabled flexible style transfer, it is often difficult to maintain the original structure of the image realistically during the reverse diffusion, especially when the referenced appearance image greatly differs from the common clothing appearance. To tackle this issue, we present a novel diffusion model-based unsupervised structure-aware transfer method to semantically generate new clothes from a given clothing image and a reference appearance image. In specific, we decouple the foreground clothing with automatically generated semantic masks by conditioned labels. And the mask is further used as guidance in the denoising process to preserve the structure information. Moreover, we use the pre-trained vision Transformer (ViT) for both appearance and structure guidance. Our experimental results show that the proposed method outperforms state-ofthe-art baseline models, generating more realistic images in the fashion design task. Code and demo can be found at https://github.com/Rem105-210/DiffFashion.

show abstract

Unsupervised Image-to-Image Translation with Generative Prior

Cited by 39 publications

References 38 publications

Multi‐Modal Face Stylization with a Generative Prior

Multi‐Modal Face Stylization with a Generative Prior

Dynamic PDGAN: discriminator-boosted knowledge distillation for StyleGANs

DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Models

Contact Info

Product

Resources

About