ManiGAN: Text-Guided Image Manipulation

Abstract

The goal of our paper is to semantically edit parts of an image matching a given text that describes desired attributes (e.g., texture, colour, and background), while preserving other contents that are irrelevant to the text. To achieve this, we propose a novel generative adversarial network (ManiGAN), which contains two key components: text-image affine combination module (ACM) and detail correction module (DCM). The ACM selects image regions relevant to the given text and then correlates the regions with corresponding semantic words for effective manipulation. Meanwhile, it encodes original image features to help reconstruct text-irrelevant contents. The DCM rectifies mismatched attributes and completes missing contents of the synthetic image. Finally, we suggest a new metric for evaluating image manipulation results, in terms of both the generation of new attributes and the reconstruction of text-irrelevant contents. Extensive experiments on the CUB and COCO datasets demonstrate the superior performance of the proposed method. Code is available at https://github.com/mrlibw/ManiGAN.

Text-Guided Style Transfer-Based Image Manipulation Using Multimodal Generative Models

“…Recently, several methods to perform image manipulation using natural language descriptions without style transfer have been increasingly proposed [30]- [33]. These methods semantically associate a part of the input image with the user's textual description.…”

“…These methods semantically associate a part of the input image with the user's textual description. Particularly, one of the state-ofthe-art methods in text-guided image manipulation, Mani-GAN [33] has shown excellent results. ManiGAN is one of the generative adversarial networks for image manipulation consisting of text-image affine combination (ACM) moldules and detail correction (DCM) modules.…”

Segmentation-Aware Text-Guided Image Manipulation

“…On the other hand, if image manipulation by the natural language description that humans generally use becomes feasible, the development of various user-friendly applications accelerates. Hence, some text-guided image manipulation methods have been proposed recently [11][12][13]. These methods modify some parts of the input image in accordance This work was partly supported by JSPS KAKENHI Grant Number JP17H01744.…”

“…However, the state-ofthe-art methods are still insufficient for accurate image manipulation (see SISGAN [11] and TAGAN [12] in Fig. 1), and even the best performing method, ManiGAN [13], modifies undesired parts such as berries and tree branches (see ManiGAN [13] in Fig. 1).…”

“…We propose a segmentation-aware text-guided image manipulation method in this paper. We newly introduce an image segmentation network into the previous generative adversarial network for image manipulation (ManiGAN [13]) to consider the background-aware bias. The image segmentation network is used to incorporate the distinction between foreground and background, which humans unconsciously do, into the training of the generative adversarial network.…”