DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse Text-to-Image Generation

Abstract

In this paper, we concentrate on the text-to-image synthesis task that aims at automatically producing perceptually realistic pictures from text descriptions. Recently, several single-stage methods have been proposed to deal with the problems of a more complicated multi-stage modular architecture. However, they often suffer from the lack-of-diversity issue, yielding similar outputs given a single textual sequence. To this end, we present an efficient and effective single-stage framework (DiverGAN) to generate diverse, plausible and semantically consistent images according to a natural-language description. DiverGAN adopts two novel word-level attention modules, i.e., a channel-attention module (CAM) and a pixel-attention module (PAM), which model the importance of each word in the given sentence while allowing the network to assign larger weights to the significant channels and pixels semantically aligning with the salient words. After that, Conditional Adaptive Instance-Layer Normalization (CAdaILN) is introduced to enable the linguistic cues from the sentence embedding to flexibly manipulate the amount of change in shape and texture, further improving visual-semantic representation and helping stabilize the training. Also, a dual-residual structure is developed to preserve more original visual features while allowing for deeper networks, resulting in faster convergence speed and more vivid details. Furthermore, we propose to plug a fully-connected layer into the pipeline to address the lack-of-diversity problem, since we observe that a dense layer will remarkably enhance the generative capability of the network, balancing the trade-off between a low-dimensional random latent code contributing to variants and modulation modules that use high-dimensional and textual contexts to strength feature maps. Inserting a linear layer after the second residual block achieves the best variety and quality. Both qualitative and quantitative results on benchmark data sets demonstrate the superiority of our DiverGAN for realizing diversity, without harming quality and semantic consistency.

Automated Voice-to-Image Generation Using Generative Adversarial Networks in Machine Learning

“…As previously said, there are several efforts on text-to-image generation [10,11,13,14]. All of these are accomplished through generative adversarial networks [1,2], as proposed by Ian Goodfellow in his paper Generative Adversarial Networks [1].…”

“…They repeatedly produced the exact output for the same comparable text. So DiverGAN [14] is developed to deal with diversity difficulties by using a fully connected layer to minimize the diversity problem.…”

“…For example, if we want to see a purple bird, we may tell the model and it will generate the image for us. This is employed by using generative adversarial networks [1], where they can create visuals that are as excellent as our words [10,11,13,14]. It takes a long time, almost days, for a single individual to sketch or produce something.…”

“…Our technology has advanced significantly and there are several text-to-image generation models [10,11,13,14] that uses text as input to produce images. Recently, speech-based work [16] has been developed that construct a person's visage based on their voice.…”