A Picture is Worth a Thousand Words

Abstract

A creative image-and-text generative AI system mimics humans' extraordinary abilities to provide users with diverse and comprehensive caption suggestions, as well as rich image creations. In this work, we demonstrate such an AI creation system to produce both diverse captions and rich images. When users imagine an image and associate it with multiple captions, our system paints a rich image to reflect all captions faithfully. Likewise, when users upload an image, our system depicts it with multiple diverse captions. We propose a unified multi-modal framework to achieve this goal. Specifically, our framework jointly models image-and-text representations with a Transformer network, which supports rich image creation by accepting multiple captions as input. We consider the relations among input captions to encourage diversity in training and adopt a non-autoregressive decoding strategy to enable realtime inference. Based on these, our system supports both diverse captions and rich images generations. Our code is available online 1 . CCS CONCEPTS• Computing methodologies → Natural language generation; Image processing.

Reject Decoding via Language-Vision Models for Text-to-Image Synthesis

“…Kim et al (Kim et al 2022) proposed the L-Verse with a feature-augmented variational autoencoder and bidirectional auto-regressive transformers for the image-text bidirectional generating. Huang et al (Huang et al 2021a) exploited a transformer to synthesize highquality images conditioned on multiple captions. Esser et al proposed the ImageBART to synthesize images in a coarse-to-fine manner by using autoregressive models and the multinomial diffusion process.…”