LAFITE: Towards Language-Free Training for Text-to-Image Generation

Abstract

One of the major challenges in training text-to-image generation models is the need of a large number of highquality image-text pairs. While image samples are often easily accessible, the associated text descriptions typically require careful human captioning, which is particularly time-and cost-consuming. In this paper, we propose the first work to train text-to-image generation models without any text data. Our method leverages the well-aligned multimodal semantic space of the powerful pre-trained CLIP model: the requirement of text-conditioning is seamlessly alleviated via generating text features from image features. Extensive experiments are conducted to illustrate the effectiveness of the proposed method. We obtain state-of-theart results in the standard text-to-image generation tasks. Importantly, the proposed language-free model outperforms most existing models trained with full image-text pairs. Furthermore, our method can be applied in fine-tuning pretrained models, which saves both training time and cost in training text-to-image generation models. Our pre-trained model obtains competitive results in zero-shot text-to-image generation on the MS-COCO dataset, yet with around only 1% of the model size and training data size relative to the recently proposed large DALL-E model.

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

“…The human evaluator may also indicate that neither image is significantly better than the other, in which case half of a win is assigned to both models. (Zhu et al, 2019) 32.64 DF-GAN (Tao et al, 2020) 21.42 DM-GAN + CL (Ye et al, 2021) 20.79 XMC-GAN (Zhang et al, 2021) 9.33 LAFITE (Zhou et al, 2021) 8.12 DALL-E (Ramesh et al, 2021) ∼ 28 LAFITE (Zhou et al, 2021) 26.94 GLIDE 12.24 GLIDE (Validation filtered)…”

“… edits images using text prompts by fine-tuning a diffusion model to target a CLIP loss while reconstructing the original image's DDIM latent Zhou et al (2021). trains GAN models conditioned on perturbed CLIP image embeddings, resulting in a model which can condition images on CLIP text embeddings.…”

CM3: A Causal Masked Multimodal Model of the Internet

“…We see that CM3 is capable of generating non-trivial semantically coherent captions. That being said, most failure cases of our proposed zero-shot captioning are due Model FID Zero-shot FID AttnGAN (Xu et al, 2017) 35.49 DM-GAN (Zhu et al, 2019) 32.64 DF-GAN (Tao et al, 2020) 21.42 DM-GAN + CL (Ye et al, 2021) 20.79 XMC-GAN 9.33 LAFITE (Zhou et al, 2021) 8.12 DALL-E ∼ 28 LAFITE (Zhou et al, 2021) 26.94 GLIDE (Nichol et al, 2021) 12. 2021) we sample roughly 30k conditioned samples for our models, and compare against the entire validation set.…”

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

“…Text-to-image generation [64,72,54,65,67,45,12,41,70] focuses on generating images from standalone text descriptions. Preliminary text-to-image methods conditioned RNN-based DRAW [18] on text [40].…”

“…Inspired by the high-quality unconditional images generation model, GLIDE employed guided inference with and without a classifier network to generate high-fidelity images. LAFITE [70] employed a pre-trained CLIP [44] model to project text and images to the same latent space, training text-to-image models without text data. Similarly to DALL-E and CogView, we train an autoregressive transformer model on text and image tokens.…”