Imagic: Text-Based Real Image Editing with Diffusion Models

Kawar, Bahjat; Zada, Shiran; Lang, Oran; Tov, Omer; Chang, Hui‐Wen; Dekel, Tali; Mosseri, Inbar; Irani, Michal

doi:10.48550/arxiv.2210.09276

Cited by 45 publications

(82 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another related line of work aims to introduce specific concepts to a pre-trained text-to-image model by learning to map a set of images to a "word" in the embedding space of the model [18,25,41]. Several works have also explored providing users with more control over the synthesis process solely through the use of the input text prompt [8,20,24,46].…”

Section: Related Workmentioning

confidence: 99%

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

Chefer¹,

Alaluf²,

Vinker³

et al. 2023

Preprint

View full text Add to dashboard Cite

Figure 1. Given a pre-trained text-to-image diffusion model (e.g. Stable Diffusion [39]) our method, Attend-and-Excite, guides the generative model to modify the cross-attention values during the image synthesis process to generate images that more faithfully depict the input text prompt. Stable Diffusion alone (top row) struggles to generate multiple objects (e.g. a horse and a dog). However, by incorporating Attend-and-Excite (bottom row) to strengthen the subject tokens (marked in blue), we achieve images that are more semantically faithful with respect to the input text prompts.

show abstract

Section: Related Workmentioning

confidence: 99%

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

Chefer¹,

Alaluf²,

Vinker³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This unprecedented capability became instantly popular, as users were able to synthesize high-quality images by simply describing the desired result in natural language, as we demonstrate in Figure 9.5. These models have become a centerpiece in an ongoing and quickly advancing research area, as they have been adapted for image editing [147,202], object recontextualization [241,95], 3D object generation [220], and more [119,129,213,346].…”

Section: Regularization By Denoising (Red)mentioning

confidence: 99%

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

Elad¹,

Kawar²,

Vaksman³

2023

Preprint

View full text Add to dashboard Cite

Image denoising -removal of additive white Gaussian noise from an image -is one of the oldest and most studied problems in image processing. An extensive work over several decades has led to thousands of papers on this subject, and to many well-performing algorithms for this task. Indeed, ten years ago, these achievements have led some researchers to suspect that "Denoising is Dead", in the sense that all that can be achieved in this domain has already been obtained. However, this turned out to be far from the truth, with the penetration of deep learning (DL) into the realm of image processing. The era of DL brought a revolution to image denoising, both by taking the lead in today's ability for noise suppression in images, and by broadening the scope of denoising problems being treated. Our paper starts by describing this evolution, highlighting in particular the tension and synergy that exist between classical approaches and modern Artificial Intelligence (AI) alternatives in design of image denoisers.The recent transitions in the field of image denoising go far beyond the ability to design better denoisers. In the second part of this paper we focus on recently discovered abilities and prospects of image denoisers. We expose the possibility of using image denoisers for service of other problems, such as regularizing general inverse problems and serving as the prime engine in diffusion-based image synthesis. We also unveil the (strange?) idea that denoising and other inverse problems might not have a unique solution, as common algorithms would have us believe. Instead, we describe constructive ways to produce randomized and diverse high perceptual quality results for inverse problems, all fueled by the progress that DL brought to image denoising. This is a survey paper, and its prime goal is to provide a broad view of the history of the field of image denoising and closely related topics in image processing. Our aim is to give a better context to recent discoveries, and to the influence of the AI revolution in our domain.

show abstract

“…To demonstrate the difficulty of eL-TGIM, we adopt several recent powerful image editors, including transformer-based architectures (Muse [1]), diffusion-based pixel-level model (Imagen [2] based Imagic [3]) and latent-level model (DALLE2 [4]). Results are shown in Fig.…”

Section: Generated Maskmentioning

confidence: 99%

“…1. Comparison between our method and powerful image editors, including transformer-based architectures (Muse [1]), diffusion-based pixel-level model (Imagen [2] based Imagic [3]) and latent-level model (DALLE2 [4]). Analyses are in the second paragraph of Sec.…”

Section: Introductionmentioning

confidence: 99%

ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation

Wang

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Existing text-guided image manipulation methods aim to modify the appearance of the image or to edit a few objects in a virtual or simple scenario, which is far from practical applications. In this work, we study a novel task on text-guided image manipulation on the entity level in the real world (eL-TGIM). The task imposes three basic requirements, (1) to edit the entity consistent with the text descriptions, (2) to preserve the entity-irrelevant regions, and (3) to merge the manipulated entity into the image naturally. To this end, we propose an elegant framework, dubbed as SeMani, forming the Semantic Manipulation of real-world images that can not only edit the appearance of entities but also generate new entities corresponding to the text guidance. To solve eL-TGIM, SeMani decomposes the task into two phases: the semantic alignment phase and the image manipulation phase. In the semantic alignment phase, SeMani incorporates a semantic alignment module to locate the entity-relevant region to be manipulated. In the image manipulation phase, SeMani adopts a generative model to synthesize new images conditioned on the entity-irrelevant regions and target text descriptions. We discuss and propose two popular generation processes that can be utilized in SeMani, the discrete auto-regressive generation with transformers and the continuous denoising generation with diffusion models, yielding SeMani-Trans and SeMani-Diff, respectively. We conduct extensive experiments on the real datasets CUB, Oxford, and COCO datasets to verify that SeMani can distinguish the entity-relevant and -irrelevant regions and achieve more precise and flexible manipulation in a zero-shot manner compared with baseline methods. Our codes and models will be released at https://github.com/Yikai-Wang/SeMani.

show abstract

Imagic: Text-Based Real Image Editing with Diffusion Models

Cited by 45 publications

References 34 publications

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation

Contact Info

Product

Resources

About