Transformer models for enhancing AttnGAN based text to image generation

Naveen, S.; Kiran, Mahreen; Indupriya, M.; Manikanta, Tadikamalla; Sudeep, P.V.

doi:10.1016/j.imavis.2021.104284

Cited by 26 publications

(13 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Attentional Generative Adversarial Network (AttnGAN) enables multi-stage, attention-driven image generation from textual description [7], [8]. AttnGAN begins with a rudimentary low-resolution image which it then refines in multiple phases to produce a final image from the natural language description.…”

Section: A Gansmentioning

confidence: 99%

“…The author of [8], presented AttnGANTRANS which consists of Attentional GAN and transformer models such as Bidirectional Encoder Representations from Transformers (BERT), GPT2, and XLNet that were capable of extracting semantic information from text descriptions more accurately than the conventional AttnGAN. Gao et al [13] proposed LD-CGAN comprised of one generator and two independent discriminators to regularize and generate 64x64 and 128x128 images.…”

Section: B Text To Image Synthesismentioning

confidence: 99%

“…A work [8] from 2021 suggested that using a pre-trained text encoder enhances the performance of text-to-image synthesis. The proposed system employs Bangla FastText [2] as pretrained text encoder.…”

Section: B Text Encodermentioning

confidence: 99%

“…Prior work related to text-to-face synthesis utilizes English text descriptions where they employ sBERT [1], Roberta [15], GPT2 and XLNet [8] etc. as text encoders.…”

Section: A Experimental Setupmentioning

confidence: 99%

See 3 more Smart Citations

Facial Image Generation from Bangla Textual Description using DCGAN and Bangla FastText

Arnob,

Rahman,

Mahmud

et al. 2023

IJACSA

View full text Add to dashboard Cite

The synthesis of facial images from textual descriptions is a relatively difficult subfield of text-to-image synthesis. It is applicable in various domains like Forensic Science, Game Development, Animation, Digital Marketing, and Metaverse. However, no work was found that generates facial images from textual descriptions in Bangla; the 5th most spoken language in the world. This research introduces the first-ever system to generate facial images from Bangla textual descriptions. The proposed model comprises two fundamental constituents, namely a textual encoder, and a Generative Adversarial Network(GAN). The text encoder is a pre-trained Bangla text encoder named Bangla FastText which is employed to transform Bangla text into a latent vector representation. The utilization of Deep Convolutional GAN (DCGAN) allows for the generation of face images that correspond to text embedding. Furthermore, a Bangla version of the CelebA dataset, CelebA Bangla is created for this study to develop the proposed system. CelebA Bangla contains images of celebrities, their corresponding annotated Bangla facial attributes and Bangla Textual Descriptions generated using a novel description generation algorithm. The proposed system attained a Fréchet Inception Distance (FID) score of 126.708, Inception Score(IS) of 12.361, and Face Semantic Distance(FSD) of 20.23. The novel text embedding strategy used in this study outperforms prior work. A thorough qualitative and quantitative analysis demonstrates the superior performance of the proposed system over other experimental systems.

show abstract

Section: A Gansmentioning

confidence: 99%

Section: B Text To Image Synthesismentioning

confidence: 99%

Section: B Text Encodermentioning

confidence: 99%

“…Prior work related to text-to-face synthesis utilizes English text descriptions where they employ sBERT [1], Roberta [15], GPT2 and XLNet [8] etc. as text encoders.…”

Section: A Experimental Setupmentioning

confidence: 99%

See 2 more Smart Citations

Facial Image Generation from Bangla Textual Description using DCGAN and Bangla FastText

Arnob,

Rahman,

Mahmud

et al. 2023

IJACSA

View full text Add to dashboard Cite

show abstract

“…Through further studies, researchers have made several improvements to the Transformer, resulting in networks such as DETR [ 24 ], ViT [ 25 ], and SETR-MLP [ 26 ]. These networks have been applied to various fields, such as object detection [ 27 , 28 , 29 ], semantic segmentation [ 30 , 31 , 32 ], image classification [ 33 , 34 , 35 ], and image generation [ 36 ]. In this study, the advanced Swin-Transformer [ 37 ] network is used to conduct research in the following three aspects: (1) Proposing a high-precision classification and detection model for mutton multi-parts; (2) testing the robustness, generalization, and anti-occlusion performance of the proposed model; (3) introducing other mainstream detection algorithms to evaluate the advantages and disadvantages of the proposed model and test its real-time performance.…”

Section: Introductionmentioning

confidence: 99%

Research on Automatic Classification and Detection of Mutton Multi-Parts Based on Swin-Transformer

Zhao

Bai

Wang

et al. 2023

Foods

View full text Add to dashboard Cite

In order to realize the real-time classification and detection of mutton multi-part, this paper proposes a mutton multi-part classification and detection method based on the Swin-Transformer. First, image augmentation techniques are adopted to increase the sample size of the sheep thoracic vertebrae and scapulae to overcome the problems of long-tailed distribution and non-equilibrium of the dataset. Then, the performances of three structural variants of the Swin-Transformer (Swin-T, Swin-B, and Swin-S) are compared through transfer learning, and the optimal model is obtained. On this basis, the robustness, generalization, and anti-occlusion abilities of the model are tested and analyzed using the significant multiscale features of the lumbar vertebrae and thoracic vertebrae, by simulating different lighting environments and occlusion scenarios, respectively. Furthermore, the model is compared with five methods commonly used in object detection tasks, namely Sparser-CNN, YoloV5, RetinaNet, CenterNet, and HRNet, and its real-time performance is tested under the following pixel resolutions: 576 × 576, 672 × 672, and 768 × 768. The results show that the proposed method achieves a mean average precision (mAP) of 0.943, while the mAP for the robustness, generalization, and anti-occlusion tests are 0.913, 0.857, and 0.845, respectively. Moreover, the model outperforms the five aforementioned methods, with mAP values that are higher by 0.009, 0.027, 0.041, 0.050, and 0.113, respectively. The average processing time of a single image with this model is 0.25 s, which meets the production line requirements. In summary, this study presents an efficient and intelligent mutton multi-part classification and detection method, which can provide technical support for the automatic sorting of mutton as well as for the processing of other livestock meat.

show abstract