DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation

Zhang, Zhenxing; Schomaker, Lambert

doi:10.1109/ijcnn52387.2021.9533527

Cited by 38 publications

(31 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tao et al [34] proposed DFGAN where a matching-aware zero-centered gradient penalty loss is introduced to help stabilize the training of the conditional text-to-image GAN model. Zhang et al [35] designed DTGAN by utilizing spatial and channel attention modules and the conditional normalization to yield photo-realistic samples with a generator/discriminator pair. Zhang et al [36] developed XMC-GAN which studied contrastive learning in the context of text-to-image generation while producing visually plausible images via a simple single-stage framework.…”

Section: Cgan In Text-to-image Generationmentioning

confidence: 99%

“…After that, the generator ( , ( , )) is trained to produce a perceptually-realistic and semanticallyrelated image ̂ according to a latent code randomly sampled from a frozen distribution and word/sentence embedding vectors ( , ). To be specific, ( , ( , )) consists of multiple layers where the first layer 0 maps a latent code into a feature map and intermediate blocks typically leverage modulation modules (e.g., attention models [35,2]) to reinforce the visual feature map to ensure image quality and semantic consistency. The last layer transforms the feature map into the ultimate sample.…”

Section: Preliminarymentioning

confidence: 99%

See 1 more Smart Citation

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

Zhang¹,

Schomaker²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Text-to-image generation intends to automatically produce a photo-realistic image, conditioned on a textual description. It can be potentially employed in the field of art creation, data augmentation, photo-editing, etc. Although many efforts have been dedicated to this task, it remains particularly challenging to generate believable, natural scenes. To facilitate the real-world applications of text-toimage synthesis, we focus on studying the following three issues: 1) How to ensure that generated samples are believable, realistic or natural? 2) How to exploit the latent space of the generator to edit a synthesized image? 3) How to improve the explainability of a text-to-image generation framework? In this work, we constructed two novel data sets (i.e., the Good & Bad bird and face data sets) consisting of successful as well as unsuccessful generated samples, according to strict criteria. To effectively and efficiently acquire high-quality images by increasing the probability of generating latent codes, we use a dedicated Good/Bad classifier for generated images. It is based on a pre-trained front end and fine-tuned on the basis of the proposed Good & Bad data set. After that, we present a novel algorithm which identifies semantically-understandable directions in the latent space of a conditional text-to-image GAN architecture by performing independent component analysis on the pre-trained weight values of the generator. Furthermore, we develop a backgroundflattening loss (BFL), to improve the background appearance in the edited image. Subsequently, we introduce linear interpolation analysis between pairs of keywords. This is extended into a similar triangular 'linguistic' interpolation in order to take a deep look into what a text-to-image synthesis model has learned within the linguistic embeddings. Experimental results on the recent Diver-GAN generator pre-trained on three benchmark data sets demonstrate that our classifier achieves a better than 98% accuracy in predicting Good/Bad classes for synthetic samples and our proposed approach is able to derive various interpretable semantic properties for a conditional text-to-image GAN model, confirming the effectiveness of our presented techniques. Our data set is available at https://zenodo.org/record/6283798#.YhkN_ujMI2w.

show abstract

Section: Cgan In Text-to-image Generationmentioning

confidence: 99%

Section: Preliminarymentioning

confidence: 99%

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

Zhang¹,

Schomaker²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Tao et al [4] designed a matchingaware zero-centered gradient penalty (MA-GP) loss addressing the issues of the multi-stage framework. Zhang et al [1] presented DTGAN leveraging two new attention models and conditional adaptive instance-layer normalization to produce perceptually realistic samples with a generator/discriminator pair. Zhang et al [5] proposed DiverGAN which can yield diverse and visually plausible samples which are semantically correlated with given natural-language descriptions.…”

Section: Related Work a Text-to-image Generationmentioning

confidence: 99%

“…) denotes a suite of K natural-language descriptions, while S is cast into a training set and a testing set. Current conditional text-to-image GANs [1]- [3] approaches commonly follow the same paradigm. The generator G aims at yielding a visually plausible and semantically consistent sample Îi according to a latent code randomly sampled from a fixed distribution and a text description c i randomly picked from C i , where c i = (w 1 , w 2 , ..., w m ) contains m words.…”

Section: A Preliminarymentioning

confidence: 99%

“…Research in conditional generative adversarial networks (cGANs) and style generative adversarial networks (GANs) has rapidly attracted much attention in recent years, since these can be potentially applied in various fields, such as art creation, computer-aid design, data augmentation for training image classifiers, photo-editing, the education of young children, etc. Existing text-to-image generation methods [1]- [4] have achieved tremendous improvements in both image quality and semantic consistency with the cGANs, mapping a latent code into an image sample semantically aligning with the input text via adversarial training. Despite the promising results, it is still difficult to explain what a conditional text-to-image GANs model has learned within the latent space and semantic space.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Optimized latent-code selection for explainable conditional text-to-image GANs

Zhang¹,

Schomaker²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The task of text-to-image generation has achieved remarkable progress due to the advances in the conditional generative adversarial networks (GANs). However, existing conditional text-to-image GANs approaches mostly concentrate on improving both image quality and semantic relevance but ignore the explainability of the model which plays a vital role in real-world applications. In this paper, we present a variety of techniques to take a deep look into the latent space and semantic space of the conditional text-to-image GANs model. We introduce pairwise linear interpolation of latent codes and 'linguistic' linear interpolation to study what the model has learned within the latent space and 'linguistic' embeddings. Subsequently, we extend linear interpolation to triangular interpolation conditioned on three corners to further analyze the model. After that, we build a Good/Bad data set containing unsuccessfully and successfully synthetic samples and corresponding latent codes for the image-quality research. Based on this data set, we propose a framework for finding good latent codes by utilizing a linear SVM. Experimental results on the recent DiverGAN generator trained on two benchmark data sets qualitatively prove the effectiveness of our presented techniques, with a better than 94% accuracy in predicting Good/Bad classes for latent vectors. The Good/Bad data set is publicly available at https://zenodo.org/record/5850224#.YeGMwP7MKUk.

show abstract

Text2Plot: Sentiment Analysis by Creating 2D Plot Representations of Texts

Dovdon

Batsuuri

2021

IEEJ Transactions Elec Engng

View full text Add to dashboard Cite

We introduce a novel approach for generating 2D RGB color images with a plot from the micro text (tweet) to be used for the overall polarity classification process of sentiment analysis. Researchers generally use word embedding and external resource‐based embedding techniques for text preprocessing of sentiment analysis through machine learning, neural networks, and natural language processing approaches. We sought to identify alternative ways to represent tweets for text classification. According to the experimental results, using the new ‘Text2Plot’ representation method could increase F1 scores by 27.2% for Convolutional neural networks (CNNs), 10.3% for support vector machine, and 4.4% for random forest models compared to using simple vectors as features for sentiment analysis. Hence, we propose this new method as a useful text representation approach for sentiment analysis, natural language processing tasks, and image processing problems. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

show abstract

DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation

Cited by 38 publications

References 20 publications

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

Optimized latent-code selection for explainable conditional text-to-image GANs

Text2Plot: Sentiment Analysis by Creating 2D Plot Representations of Texts

Contact Info

Product

Resources

About