An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Abstract

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion. github.io → Input samples invert −−−−→ "S * " "An oil painting of S * " "App icon of S * " "Elmo sitting in the same pose as S * " "Crochet S * " → Input samples invert −−−−→ "S * " "Painting of two S * fishing on a boat" "A S * backpack" "Banksy art of S * " "A S * themed lunchbox"Preprint. Under review.

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

“…Another related line of work aims to introduce specific concepts to a pre-trained text-to-image model by learning to map a set of images to a "word" in the embedding space of the model [18,25,41]. Several works have also explored providing users with more control over the synthesis process solely through the use of the input text prompt [8,20,24,46].…”

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

“…This unprecedented capability became instantly popular, as users were able to synthesize high-quality images by simply describing the desired result in natural language, as we demonstrate in Figure 9.5. These models have become a centerpiece in an ongoing and quickly advancing research area, as they have been adapted for image editing [147,202], object recontextualization [241,95], 3D object generation [220], and more [119,129,213,346].…”

Content-Based Search for Deep Generative Models

“…Transfer learning aims to adapt models to unseen domains and tasks [47,61,87,102,128,131]. For generative models like GANs and diffusion models, several works have proposed finetuning a pre-trained network to enable image generation in an unseen limited-data domain [33,66,77,81,82,101,124,125,134]. Various works have explored model selection to choose pre-trained models for discriminative tasks [12,27,80,93] or selecting pre-trained discriminators for training GANs [63].…”

“…Each model captures a small universe of curated subjects, which can range from realistic rendering of faces and landscapes [56] to photos of historic pottery [5] to cartoon caricatures [50] to single-artist stylistic elements [108]. More recently, various methods enable creative modifications and personalization of existing models, via human-in-the-loop interfaces [8,34,122,123] or fine-tuning of GANs [83,125,135] and text-to-image models [33,101]. Each generative model can represent a substantial investment in a specific idea of the model creator.…”