Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis

Tang, Hao; Qi, Xiaojuan; Guolei, Sun,; Xu, Dianguo; Sebe, Nicu; Timofte, Radu; Gool, Luc Van

doi:10.48550/arxiv.2003.13898

Cited by 11 publications

(10 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other authors addressed the problem by using both categorical information of the training images and also their semantic segmentation, as in [6,7,17,18]. In those works, the authors train the discriminator to distinguish real and fake images and at the same time, to match the objective pixel wise semantic information given.…”

Section: In This Context Authors Have Developed Different Approaches ...mentioning

confidence: 99%

“…This gives rise to conditional GANs, which are able to generate artificial images with certain specific, desired characteristics. Many authors in recent years have explored different ways of including this conditional information, be it through image textual descriptions [5], semantic segmentations [6] or image category definition [7], among others. However, all these approaches set the number and definition of characteristics before the GAN training process and cannot be changed later.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Guiding GANs: How to control non-conditional pre-trained GANs for conditional image generation

Mateos,

González,

Sevillano

2021

Preprint

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) are an arrange of two neural networks -the generator and the discriminator-that are jointly trained to generate artificial data, such as images, from random inputs. The quality of these generated images has recently reached such levels that can often lead both machines and humans into mistaking fake for real examples. However, the process performed by the generator of the GAN has some limitations when we want to condition the network to generate images from subcategories of a specific class. Some recent approaches tackle this conditional generation by introducing extra information prior to the training process, such as image semantic segmentation or textual descriptions. While successful, these techniques still require defining beforehand the desired subcategories and collecting large labeled image datasets representing them to train the GAN from scratch. In this paper we present a novel and alternative method for guiding generic non-conditional GANs to behave as conditional GANs. Instead of re-training the GAN, our approach adds into the mix an encoder network to generate the high-dimensional random input vectors that are fed to the generator network of a non-conditional GAN to make it generate images from a specific subcategory. In our experiments, when compared to training a conditional GAN from scratch, our guided GAN is able to generate artificial images of perceived quality comparable to that of non-conditional GANs after training the encoder on just a few hundreds of images, which substantially accelerates the process and enables adding new subcategories seamlessly.

show abstract

Section: In This Context Authors Have Developed Different Approaches ...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Guiding GANs: How to control non-conditional pre-trained GANs for conditional image generation

Mateos,

González,

Sevillano

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides, the quality of image-to-image translation has been significantly improved by some recent works [97], [115], [148], [184], [207], [238]. In particular, pix2pixHD [207] is able to generate high-resolution images with a coarse-tofine generator and a multi-scale discriminator.…”

Section: Supervised Image Translationmentioning

confidence: 99%

Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications

Huang

et al. 2020

Preprint

View full text Add to dashboard Cite

The generative adversarial network (GAN) framework has emerged as a powerful tool for various image and video synthesis tasks, allowing the synthesis of visual content in an unconditional or input-conditional manner. It has enabled the generation of high-resolution photorealistic images and videos, a task that was challenging or impossible with prior methods. It has also led to the creation of many new applications in content creation. In this paper, we provide an overview of GANs with a special focus on algorithms and applications for visual synthesis. We cover several important techniques to stabilize GAN training, which has a reputation for being notoriously difficult. We also discuss its applications to image translation, image processing, video synthesis, and neural rendering.

show abstract

“…Most recently, LGGAN [56] focused on improving synthesis of small objects and local details, by designing a generator with separate branches that jointly learn the local class-specific and global image-level generation. Likewise, the authors in [55] improve the synthesis of local structures via an attention-based edge guided generator network. Besides GANs, CRN [8] and SIMS [42] train a cascaded refinement convolutional network with a regression loss.…”

Section: Related Workmentioning

confidence: 99%

Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis

Katiyar,

Khoreva

2020

Preprint

View full text Add to dashboard Cite

Despite data augmentation being a de facto technique for boosting the performance of deep neural networks, little attention has been paid to developing augmentation strategies for generative adversarial networks (GANs). To this end, we introduce a novel augmentation scheme designed specifically for GAN-based semantic image synthesis models. We propose to randomly warp object shapes in the semantic label maps used as an input to the generator. The local shape discrepancies between the warped and nonwarped label maps and images enable the GAN to learn better the structural and geometric details of the scene and thus to improve the quality of generated images. While benchmarking the augmented GAN models against their vanilla counterparts, we discover that the quantification metrics reported in the previous semantic image synthesis studies are strongly biased towards specific semantic classes as they are derived via an external pre-trained segmentation network. We therefore propose to improve the established semantic image synthesis evaluation scheme by analyzing separately the performance of generated images on the biased and unbiased classes for the given segmentation network. Finally, we show strong quantitative and qualitative improvements obtained with our augmentation scheme, on both class splits, using state-of-the-art semantic image synthesis models across three different datasets. On average across COCO-Stuff, ADE20K and Cityscapes datasets, the augmented models outperform their vanilla counterparts by ∼3 mIoU and ∼10 FID points.

show abstract

Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis

Cited by 11 publications

References 50 publications

Guiding GANs: How to control non-conditional pre-trained GANs for conditional image generation

Guiding GANs: How to control non-conditional pre-trained GANs for conditional image generation

Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications

Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis

Contact Info

Product

Resources

About