StoryGAN: A Sequential Conditional GAN for Story Visualization

Li, Yitong; Gan, Zhe; Shen, Yulong; Liu, Jingjing; Cheng, Yu; Wu, Yuexin; Carin, Lawrence; Carlson, David; Gao, Jianfeng

doi:10.1109/cvpr.2019.00649

Cited by 144 publications

(144 citation statements)

References 35 publications

Supporting

Mentioning

133

Contrasting

Order By: Relevance

“…Progressive Manipulation A few state of the art approaches for text-to-image generation and image manipulation are expressly designed for conversational systems [5,8,9,21,30]. In Figure 5 we show that we can repeatedly apply our method to generated images to have a manipulation in multiples steps.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Describe What to Change

Liu

Nadai

Cai

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to use richly-annotated image captioning datasets. In this work, we propose a novel unsupervised approach, based on imageto-image translation, that alters the attributes of a given image through a command-like sentence such as "change the hair color to black". Contrarily to state-of-the-art approaches, our model does not require a human-annotated dataset nor a textual description of all the attributes of the desired image, but only those that have to be modified. Our proposed model disentangles the image content from the visual attributes, and it learns to modify the latter using the textual description, before generating a new image from the content and the modified attribute representation. Because text might be inherently ambiguous (blond hair may refer to different shadows of blond, e.g. golden, icy, sandy), our method generates multiple stochastic versions of the same translation. Experiments show that the proposed model achieves promising performances on two large-scale public datasets: CelebA and CUB. We believe our approach will pave the way to new avenues of research combining textual and speech commands with visual attributes. CCS CONCEPTS • Computing methodologies → Computer vision; Machine learning.

show abstract

Section: Resultsmentioning

confidence: 99%

“…proposed StoryGAN [21], which generates a series of images that are contextually coherent with previously generated images and with the sequence of text descriptions provided by the user.…”

Section: Input Text Featuresmentioning

confidence: 99%

Describe What to Change

Liu

Nadai

Cai

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…[39] allow users to input object instance masks into an existing image represented by a semantic layout. [40] generate images iteratively from consecutive textual commands, [41] provide interactive image editing based on a current image and instructions on how to update the image, and [42] generate individual images for a sequence of sentences. [43] do interactive image generation but do not use text as direct input but instead update a scene graph from text over the course of the interaction.…”

Section: Related Workmentioning

confidence: 99%

Semantic Object Accuracy for Generative Text-to-Image Synthesis

Hinz

Heinrich

Wermter

2022

IEEE Trans. Pattern Anal. Mach. Intell.

106

View full text Add to dashboard Cite

Generative adversarial networks conditioned on textual image descriptions are capable of generating realistic-looking images. However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain. Furthermore, quantitatively evaluating these text-to-image models is challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption. To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption. The SOA uses a pre-trained object detector to evaluate if a generated image contains objects that are mentioned in the image caption, e.g. whether an image generated from "a car driving down the street" contains a car. We perform a user study comparing several text-to-image models and show that our SOA metric ranks the models the same way as humans, whereas other metrics such as the Inception Score do not. Our evaluation also shows that models which explicitly model objects outperform models which only model global image characteristics.

show abstract

“…Dialogue based interaction is studied to control image synthesis, in order to improve complex scene generation progressively [219]- [223]. Meanwhile, text-to-image synthesis is extended to multiple images or videos, where visual consistency is required among the generated images [224]- [226].…”

Section: ) Other Topicsmentioning

confidence: 99%

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Zhang

Yang

et al. 2020

IEEE J. Sel. Top. Signal Process.

282

View full text Add to dashboard Cite

Deep learning has revolutionized speech recognition, image recognition, and natural language processing since 2010, each involving a single modality in the input signal. However, many applications in artificial intelligence involve more than one modality. It is therefore of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of vision and natural language, which has become an important area in both computer vision and natural language processing research communities.This review provides a comprehensive analysis of recent work on multimodal deep learning from three new angles -learning multimodal representations, the fusion of multimodal signals at various levels, and multimodal applications. On multimodal representation learning, we review the key concept of embedding, which unifies the multimodal signals into the same vector space and thus enables cross-modality signal processing. We also review the properties of the many types of embedding constructed and learned for general downstream tasks. On multimodal fusion, this review focuses on special architectures for the integration of the representation of unimodal signals for a particular task. On applications, selected areas of a broad interest in current literature are covered, including caption generation, text-to-image generation, and visual question answering. We believe this review can facilitate future studies in the emerging field of multimodal intelligence for the community.

show abstract

StoryGAN: A Sequential Conditional GAN for Story Visualization

Cited by 144 publications

References 35 publications

Describe What to Change

Describe What to Change

Semantic Object Accuracy for Generative Text-to-Image Synthesis

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Contact Info

Product

Resources

About