Realistic Image Generation from Text by Using BERT-Based Embedding

Abstract

Recently, in the field of artificial intelligence, multimodal learning has received a lot of attention due to expectations for the enhancement of AI performance and potential applications. Text-to-image generation, which is one of the multimodal tasks, is a challenging topic in computer vision and natural language processing. The text-to-image generation model based on generative adversarial network (GAN) utilizes a text encoder pre-trained with image-text pairs. However, text encoders pre-trained with image-text pairs cannot obtain rich information about texts not seen during pre-training, thus it is hard to generate an image that semantically matches a given text description. In this paper, we propose a new text-to-image generation model using pre-trained BERT, which is widely used in the field of natural language processing. The pre-trained BERT is used as a text encoder by performing fine-tuning with a large amount of text, so that rich information about the text is obtained and thus suitable for the image generation task. Through experiments using a multimodal benchmark dataset, we show that the proposed method improves the performance over the baseline model both quantitatively and qualitatively.

Multi-View Video Quality Enhancement Method Based on Multi-Scale Fusion Convolutional Neural Network and Visual Saliency

“…Since then, CNN has largely adopted the LeNet architecture, which consists of three main components: a convolution layer, a pooling layer, and a FC layer. Feature graphs are obtained through convolution and pooling operations, and they are then transformed into one-dimensional vectors and input into the FC layer, where two or more classifications are achieved through the classification layer [13]. This section provides a brief explanation of the principle and purpose of each CNN component, using the structure of LeNet as an example [14].…”

Deep neural architecture for natural language image synthesis for Tamil text using BASEGAN and hybrid super resolution GAN (HSRGAN)

“…This inspires us to improve BERT’s NMT use. The BERT-fused model first uses BERT to uproot depictions for an input sequence, then uses attention mechanisms to fuse the representations with every layer of encoder and decoder of the NMT model 9 , 10 .…”