DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse Text-to-Image Generation

Zhang, Zhenxing; Schomaker, Lambert

doi:10.1016/j.neucom.2021.12.005

Cited by 19 publications

(22 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We therefore carry out a serious of experiments on the DiverGAN generator that was trained on three popular text-to-image data sets (i.e., the CUB bird [3], MS COCO [12] and Multi-Modal CelebA-HQ [4] data sets). The experimental results in the current study represent an improvement in performance and explainability in the analyzed algorithm [2]. Meanwhile, our well-trained classifier achieves impressive classification accuracy (bird: 98.09% and face: 99.16%) on the Good & Bad data set and our proposed semantic-discovery algorithm can lead to a more precise control over the latent space of the DiverGAN model, which validate the effectiveness of our presented methods.…”

Section: Introductionsupporting

confidence: 71%

“…Zhang et al [36] developed XMC-GAN which studied contrastive learning in the context of text-to-image generation while producing visually plausible images via a simple single-stage framework. Zhang et al [2] presented an efficient and effective single-stage framework called Diver-GAN which is capable of generating diverse, plausible and semantically-consistent images according to a natural-language description. Note that we adopt the DiverGAN generator to perform comprehensive experiments due to its superior performance on image quality and diversity.…”

Section: Cgan In Text-to-image Generationmentioning

confidence: 99%

“…To this end, we created a Good & Bad data set, both for a bird and a faceimage collection (shown in Fig. 3), which consists of a large number of realistic as well as implausible samples synthesized by the recent DiverGAN [2] that was pretrained on the CUB bird [3] data set and the Multi-Modal CelebA-HQ data set [4], respectively. We choose these samples by following strict principles in order to ensure the quality of the selected images.…”

Section: Introductionmentioning

confidence: 99%

“…To acquire a superior classifier, we train the CNN model (e.g., ResNet [5]) from the pre-trained weights on our Good & Bad data set. We expect that the well-trained network can correctly predict the quality class of synthesized Interpretable latent-space directions identified in DiverGAN [2] that was pre-trained on the CUB bird [3] (left side) and Multi-Modal CelebA-HQ [4] (right side) data sets. For each set of pictures, the middle column is the original image based on a latent code, while the samples on the left and right of it are the output by freezing the textual description and moving the latent vector backward and forward from the center, over the axis discovered by our proposed algorithm.…”

Section: Introductionmentioning

confidence: 99%

“…The recent DiverGAN [2] has the ability to adopt a generator/discriminator pair to synthesize diverse and high-quality samples, given a textual description and different injected noise on the latent vector. We therefore carry out a serious of experiments on the DiverGAN generator that was trained on three popular text-to-image data sets (i.e., the CUB bird [3], MS COCO [12] and Multi-Modal CelebA-HQ [4] data sets).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

Zhang¹,

Schomaker²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Text-to-image generation intends to automatically produce a photo-realistic image, conditioned on a textual description. It can be potentially employed in the field of art creation, data augmentation, photo-editing, etc. Although many efforts have been dedicated to this task, it remains particularly challenging to generate believable, natural scenes. To facilitate the real-world applications of text-toimage synthesis, we focus on studying the following three issues: 1) How to ensure that generated samples are believable, realistic or natural? 2) How to exploit the latent space of the generator to edit a synthesized image? 3) How to improve the explainability of a text-to-image generation framework? In this work, we constructed two novel data sets (i.e., the Good & Bad bird and face data sets) consisting of successful as well as unsuccessful generated samples, according to strict criteria. To effectively and efficiently acquire high-quality images by increasing the probability of generating latent codes, we use a dedicated Good/Bad classifier for generated images. It is based on a pre-trained front end and fine-tuned on the basis of the proposed Good & Bad data set. After that, we present a novel algorithm which identifies semantically-understandable directions in the latent space of a conditional text-to-image GAN architecture by performing independent component analysis on the pre-trained weight values of the generator. Furthermore, we develop a backgroundflattening loss (BFL), to improve the background appearance in the edited image. Subsequently, we introduce linear interpolation analysis between pairs of keywords. This is extended into a similar triangular 'linguistic' interpolation in order to take a deep look into what a text-to-image synthesis model has learned within the linguistic embeddings. Experimental results on the recent Diver-GAN generator pre-trained on three benchmark data sets demonstrate that our classifier achieves a better than 98% accuracy in predicting Good/Bad classes for synthetic samples and our proposed approach is able to derive various interpretable semantic properties for a conditional text-to-image GAN model, confirming the effectiveness of our presented techniques. Our data set is available at https://zenodo.org/record/6283798#.YhkN_ujMI2w.

show abstract

Section: Introductionsupporting

confidence: 71%

Section: Cgan In Text-to-image Generationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

Zhang¹,

Schomaker²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

An Improved AttnGAN Model for Text-to-Image Synthesis

Gopalakrishnan,

Sambagni,

Sudeep

2024

Communications in Computer and Information Science

View full text Add to dashboard Cite

Optimizing and interpreting the latent space of the conditional text-to-image GANs

Zhang,

Schomaker

2023

Neural Comput & Applic

View full text Add to dashboard Cite

Text-to-image generation intends to automatically produce a photo-realistic image, conditioned on a textual description. To facilitate the real-world applications of text-to-image synthesis, we focus on studying the following three issues: (1) How to ensure that generated samples are believable, realistic or natural? (2) How to exploit the latent space of the generator to edit a synthesized image? (3) How to improve the explainability of a text-to-image generation framework? We introduce two new data sets for benchmarking, i.e., the Good & Bad, bird and face, data sets consisting of successful as well as unsuccessful generated samples. This data set can be used to effectively and efficiently acquire high-quality images by increasing the probability of generating Good latent codes with a separate, new classifier. Additionally, we present a novel algorithm which identifies semantically understandable directions in the latent space of a conditional text-to-image GAN architecture by performing independent component analysis on the pre-trained weight values of the generator. Furthermore, we develop a background-flattening loss (BFL), to improve the background appearance in the generated images. Subsequently, we introduce linear-interpolation analysis between pairs of text keywords. This is extended into a similar triangular ‘linguistic’ interpolation. The visual array of interpolation results gives users a deep look into what the text-to-image synthesis model has learned within the linguistic embeddings. Experimental results on the recent DiverGAN generator, pre-trained on three common benchmark data sets demonstrate that our classifier achieves a better than 98% accuracy in predicting Good/Bad classes for synthetic samples and our proposed approach is able to derive various interpretable semantic properties for the text-to-image GAN model.

show abstract

DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse Text-to-Image Generation

Cited by 19 publications

References 59 publications

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

An Improved AttnGAN Model for Text-to-Image Synthesis

Optimizing and interpreting the latent space of the conditional text-to-image GANs

Contact Info

Product

Resources

About