Towards Open-World Text-Guided Face Image Generation and Manipulation

Abstract

The existing text-guided image synthesis methods can only produce limited quality results with at most 256 2 resolution and the textual instructions are constrained in a small Corpus. In this work, we propose a unified framework for both face image generation and manipulation that produces diverse and high-quality images with an unprecedented resolution at 1024 2 from multimodal inputs. More importantly, our method supports open-world scenarios, including both image and text, without any re-training, fine-tuning, or post-processing. To be specific, we propose a brand new paradigm of text-guided image generation and manipulation based on the superior characteristics of a pretrained GAN model. Our proposed paradigm includes two novel strategies. The first strategy is to train a text encoder to obtain latent codes that align with the hierarchically semantic of the aforementioned pretrained GAN model. The second strategy is to directly optimize the latent codes in the latent space of the pretrained GAN model with guidance from a pretrained language model. The latent codes can be randomly sampled from a prior distribution or inverted from a given image, which provides inherent supports for both image generation and manipulation from multi-modal inputs, such as sketches or semantic labels, with textual guidance. To facilitate text-guided multi-modal synthesis, we propose the MULTI-MODAL CELEBA-HQ, a large-scale dataset consisting of real face images and corresponding semantic segmentation map, sketch, and textual descriptions. Extensive experiments on the introduced dataset demonstrate the superior performance of our proposed method.

Blended Diffusion for Text-driven Editing of Natural Images

“…It is said that "a picture is worth a thousand words", but recent research indicates that only a few words are often sufficient to describe one. Recent works that leverage the tremendous progress in vision-language models and datadriven image generation have demonstrated that text-based interfaces for image creation and manipulation are now finally within reach [12,24,29,30,39,40,42,48,53,59].…”

AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation

“…But because of the domain limitation of the existing generation models, the images they generate are limited in certain domains. Xia et al [34] map input text to StyleGAN latent space, while Xia et al [35] use cosine similarity of text and image embeddings encoded by CLIP as a loss function to optimize an embedding in StyleGAN latent space. Due to the usage of CLIP, Xia et al [35] can process texts with more complex semantics.…”

“…Xia et al [34] map input text to StyleGAN latent space, while Xia et al [35] use cosine similarity of text and image embeddings encoded by CLIP as a loss function to optimize an embedding in StyleGAN latent space. Due to the usage of CLIP, Xia et al [35] can process texts with more complex semantics. But its performance is random and visually unpleasant.…”

“…That means it is nontrivial for them to translate raw descriptions in our task. Xia et al [35] and Patashnik et al [21], have the ability to deal with complex texts, while they fail in some cases (e.g., text contains rich semantics, as shown in Figure 5) because their optimization methods cannot take full advantage of pre-trained vision-language models. Besides, optimization methods are also time-consuming so it is unpractical to widely utilize these methods.…”

DE-Net: Dynamic Text-guided Image Editing Adversarial Networks

“…They tokenize the images and take the image tokens and word tokens to make auto-regressive training by a unidirectional transformer [1,18]. [17,27,28]. They employ a pretrained StyleGAN [8] and manipulate its latent according to face descriptions.…”