Text2LIVE: Text-Driven Layered Image and Video Editing

Bar-Tal, Omer; Ofri-Amar, Dolev; Fridman, Rafail; Kasten, Yoni; Dekel, Tali

doi:10.48550/arxiv.2204.02491

Cited by 9 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While most works that require only text (i.e., no masks) are limited to global editing [9,23], Bar-Tal et al [6] proposed a text-based localized editing technique without using any mask, showing impressive results. Yet, their techniques mainly allow changing textures, but not modifying complex structures, such as changing a bicycle to a car.…”

Section: Fixed Attention Maps and Random Seedmentioning

confidence: 99%

Prompt-to-Prompt Image Editing with Cross Attention Control

Hertz¹,

Mokady²,

Tenenbaum³

et al. 2022

Preprint

126

View full text Add to dashboard Cite

Recent large-scale text-driven synthesis models have attracted much attention thanks to their remarkable capabilities of generating highly diverse images that follow given text prompts. Such text-based synthesis methods are particularly appealing to humans who are used to verbally describe their intent. Therefore, it is only natural to extend the text-driven image synthesis to text-driven image editing. Editing is challenging for these generative models, since an innate property of an editing technique is to preserve most of the original image, while in the text-based models, even a small modification of the text prompt often leads to a completely different outcome. State-of-the-art methods mitigate this by requiring the users to provide a spatial mask to localize the edit, hence, ignoring the original structure and content within the masked region. In this paper, we pursue an intuitive prompt-toprompt editing framework, where the edits are controlled by text only. To this end, we analyze a text-conditioned model in depth and observe that the cross-attention layers are the key to controlling the relation between the spatial layout of the image to each word in the prompt. With this observation, we present several applications which monitor the image synthesis by editing the textual prompt only. This includes localized editing by replacing a word, global editing by adding a specification, and even delicately controlling the extent to which a word is reflected in the image. We present our results over diverse images and prompts, demonstrating high-quality synthesis and fidelity to the edited prompts.* Performed this work while working at Google.

show abstract

Section: Fixed Attention Maps and Random Seedmentioning

confidence: 99%

Prompt-to-Prompt Image Editing with Cross Attention Control

Hertz¹,

Mokady²,

Tenenbaum³

et al. 2022

Preprint

126

View full text Add to dashboard Cite

show abstract

“…Moving beyond pure image generation, a large body of work explores the use of text-based interfaces for image editing Abdal et al, 2021;Avrahami et al, 2022b), generator domain adaptation , video manipulation (Tzaban et al, 2022;Bar-Tal et al, 2022), motion synthesis (Tevet et al, 2022;Petrovich et al, 2022), style transfer (Kwon & Ye, 2021; and even texture synthesis for 3D objects (Michel et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Gal¹,

Alaluf²,

Atzmon³

et al. 2022

Preprint

111

144

View full text Add to dashboard Cite

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion. github.io → Input samples invert −−−−→ "S * " "An oil painting of S * " "App icon of S * " "Elmo sitting in the same pose as S * " "Crochet S * " → Input samples invert −−−−→ "S * " "Painting of two S * fishing on a boat" "A S * backpack" "Banksy art of S * " "A S * themed lunchbox"Preprint. Under review.

show abstract

“…Text2LIVE [Bar-Tal et al 2022] proposed a way to edit the appearance of an object within an image, without the need to rely on a pretrained generative model. They mainly focus on changing the colors/textures of an existing object or adding effects such as fire/smoke, and not on editing a general scene by removing objects or replacing them with new ones, as we do.…”

Section: Related Workmentioning

confidence: 99%