Language-Guided Global Image Editing via Cross-Modal Cyclic Mechanism

Jiang, Wentao; Xu, Ning; Wang, Jiayun; Gao, Chen; Shi, Jing; Lin, Zhe; Liu, Si

doi:10.1109/iccv48922.2021.00212

Cited by 19 publications

(14 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…faces), which is hard to train for less structured domains like full body images. Recently, Jiang et al [37] propose a new language-guided editing model specifically designed for performing global edits such as changing brightness or color tone of an image. Language based video manipulation.…”

Section: Related Workmentioning

confidence: 99%

Disentangling Content and Motion for Text-Based Neural Video Manipulation

Karacan¹,

Kerimoğlu²,

İnan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Giving machines the ability to imagine possible new objects or scenes from linguistic descriptions and produce their realistic renderings is arguably one of the most challenging problems in computer vision. Recent advances in deep generative models have led to new approaches that give promising results towards this goal. In this paper, we introduce a new method called DiCoMoGAN for manipulating videos with natural language, aiming to perform local and semantic edits on a video clip to alter the appearances of an object of interest. Our GAN architecture allows for better utilization of multiple observations by disentangling content and motion to enable controllable semantic edits. To this end, we introduce two tightly coupled networks: (i) a representation network for constructing a concise understanding of motion dynamics and temporally invariant content, and (ii) a translation network that exploits the extracted latent content representation to actuate the manipulation according to the target description. Our qualitative and quantitative evaluations demonstrate that DiCoMoGAN significantly outperforms existing frame-based methods, producing temporally coherent and semantically more meaningful results.

show abstract

Section: Related Workmentioning

confidence: 99%

Disentangling Content and Motion for Text-Based Neural Video Manipulation

Karacan¹,

Kerimoğlu²,

İnan³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Computational approaches [5,9,30] for modifying the style and appearance of objects in natural photographs have made remarkable progress, allowing beginner users to accomplish a wide range of editing effects. Nevertheless, it should be noted that prior text-based image manipulations work either did not allow for arbitrary text commands or image manipulation [5,25,34] or only allowed for modifications to the image's appearance properties or style [20,30]. Controlling the localization of modifications normally requires the user to draw a region to specify [1,2,5], which adds to the complexity of the operation.…”

Section: Introductionmentioning

confidence: 99%

Region-Aware Diffusion for Zero-shot Text-driven Image Editing

Nisha¹,

Tang²,

Dong³

et al. 2023

Preprint

View full text Add to dashboard Cite

Fig. 1. The results of the proposed region-aware diffusion model (RDM). The texts adhere to the phrase rule "A → B", indicating that RDM transforms entity A into entity B.Image manipulation under the guidance of textual descriptions has recently received a broad range of attention. In this study, we focus on the regional editing of images with the guidance of given text prompts. Different from current mask-based image editing methods, we propose a novel region-aware diffusion model (RDM) for entity-level image editing, which could automatically locate the region of interest and replace it following given text prompts. To strike a balance between image fidelity and inference speed, we design the intensive diffusion pipeline by combing latent space diffusion and enhanced directional guidance. In addition, to preserve

show abstract

“…T HERE are various active branches of image manipulation, such as style transfer [5], image translation [6], [7], and Text-Guided Image Manipulation (TGIM), by taking advantage of recent deep generative architectures such as GANs [8], VAEs [9], auto-regressive models [10] and diffusion models [11]. Particularly, the previous TGIM methods either operate some objects by text instructions [12]- [14], such as "adding" and "removing" in a simple toy scene, or manipulating the appearance of objects [15] or the style of the image [16], [17]. In this work, we are interested in a novel challenging task of entity-Level Text-Guided Image Manipulation (eL-TGIM), which is to manipulate the entities on a natural image given the text descriptions, as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation

Wang

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Existing text-guided image manipulation methods aim to modify the appearance of the image or to edit a few objects in a virtual or simple scenario, which is far from practical applications. In this work, we study a novel task on text-guided image manipulation on the entity level in the real world (eL-TGIM). The task imposes three basic requirements, (1) to edit the entity consistent with the text descriptions, (2) to preserve the entity-irrelevant regions, and (3) to merge the manipulated entity into the image naturally. To this end, we propose an elegant framework, dubbed as SeMani, forming the Semantic Manipulation of real-world images that can not only edit the appearance of entities but also generate new entities corresponding to the text guidance. To solve eL-TGIM, SeMani decomposes the task into two phases: the semantic alignment phase and the image manipulation phase. In the semantic alignment phase, SeMani incorporates a semantic alignment module to locate the entity-relevant region to be manipulated. In the image manipulation phase, SeMani adopts a generative model to synthesize new images conditioned on the entity-irrelevant regions and target text descriptions. We discuss and propose two popular generation processes that can be utilized in SeMani, the discrete auto-regressive generation with transformers and the continuous denoising generation with diffusion models, yielding SeMani-Trans and SeMani-Diff, respectively. We conduct extensive experiments on the real datasets CUB, Oxford, and COCO datasets to verify that SeMani can distinguish the entity-relevant and -irrelevant regions and achieve more precise and flexible manipulation in a zero-shot manner compared with baseline methods. Our codes and models will be released at https://github.com/Yikai-Wang/SeMani.

show abstract

Language-Guided Global Image Editing via Cross-Modal Cyclic Mechanism

Cited by 19 publications

References 15 publications

Disentangling Content and Motion for Text-Based Neural Video Manipulation

Disentangling Content and Motion for Text-Based Neural Video Manipulation

Region-Aware Diffusion for Zero-shot Text-driven Image Editing

ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation

Contact Info

Product

Resources

About