Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model

Xu, Zipeng; Lin, Tianwei; Tang, Hao; Li, Fu; He, Dongliang; Sebe, Nicu; Timofte, Radu; Gool, Luc Van; Ding, Errui

doi:10.1109/cvpr52688.2022.01769

Cited by 29 publications

(24 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhang et al [115] propose a disentangled sentiment representation adversarial network (DiSRAN) to reduce the domain shift of expressive styles for cross-domain sentiment analysis. Recent works [116], [117], [118], [119], [120] tend to focus on disentangling the rich information among multi modalities and leveraging that to perform various downstream tasks. Alaniz et al [116] propose to use the semantic structure of the text to disentangle the visual data, in order to learn an unified representation between the text and image.…”

Section: Multimodal Applicationmentioning

confidence: 99%

“…Alaniz et al [116] propose to use the semantic structure of the text to disentangle the visual data, in order to learn an unified representation between the text and image. The PPE framework [117] realizes disentangled text-driven image manipulation through exploiting the power of the pretrained vision-language model CLIP [121]. Similarly, Yu et al [118] achieve counterfactual image manipulation via disentangling and leveraging the semantic in text embedding of CLIP.…”

Section: Multimodal Applicationmentioning

confidence: 99%

“…Multimodal Application [113], [115] VAE-based Supervised [118] GAN-based [117], [119], [120] Others [26], [28], [123], [124], [125] VAE-based Supervised Recommendation [27], [29], [122] Others [128], [129], [130], [131] VAE-based Supervised Graph [29], [30], [132] Others tation z, which is fine-grained and ii) vector-wise: use two or more independent vectors z 1 , z 2 ... to represent different parts of data features, which is coarse-grained. To guarantee the disentanglement property, approach i) usually requires that z is dimension-wise independent, while approach ii) usually requires that z i is independent with z j where i = j.…”

Section: Papersmentioning

confidence: 99%

See 2 more Smart Citations

Disentangled Representation Learning for Recommendation

Wang

Chen

Zhou

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Disentangled Representation Learning (DRL) aims to learn a model capable of identifying and disentangling the underlying factors hidden in the observable data in representation form. The process of separating underlying factors of variation into variables with semantic meaning benefits in learning explainable representations of data, which imitates the meaningful understanding process of humans when observing an object or relation. As a general learning strategy, DRL has demonstrated its power in improving the model explainability, controlability, robustness, as well as generalization capacity in a wide range of scenarios such as computer vision, natural language processing, data mining etc. In this article, we comprehensively review DRL from various aspects including motivations, definitions, methodologies, evaluations, applications and model designs. We discuss works on DRL based on two well-recognized definitions, i.e., Intuitive Definition and Group Theory Definition. We further categorize the methodologies for DRL into four groups, i.e., Traditional Statistical Approaches, Variational Auto-encoder Based Approaches, Generative Adversarial Networks Based Approaches, Hierarchical Approaches and Other Approaches. We also analyze principles to design different DRL models that may benefit different tasks in practical applications. Finally, we point out challenges in DRL as well as potential research directions deserving future investigations. We believe this work may provide insights for promoting the DRL research in the community.

show abstract

Section: Multimodal Applicationmentioning

confidence: 99%

Section: Multimodal Applicationmentioning

confidence: 99%

Section: Papersmentioning

confidence: 99%

See 1 more Smart Citation

Disentangled Representation Learning for Recommendation

Wang

Chen

Zhou

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…To generate some specific kind of images given the input for users' goal, Conditional GAN (CGAN) [23] is proposed. A CGAN always combines a basic GAN and an external information, such as labels [24], [25], [26], text descriptions [14], [27], [28], segmentation maps [29], [30], [31], [32], [33], [34], [35], [36], [37], [38] and images [8], [39]. For example, GANmut [24] introduces a novel GAN-based framework that learns an expressive and interpretable conditional space to generate a gamut of emotions, using only the categorical emotion labels.…”

Section: Related Workmentioning

confidence: 99%

Facial Expression Translation Using Landmark Guided GANs

Tang

Sebe

2022

IEEE Trans. Affective Comput.

Self Cite

View full text Add to dashboard Cite

We propose a simple yet powerful Landmark guided Generative Adversarial Network (LandmarkGAN) for the facial expressionto-expression translation using a single image, which is an important and challenging task in computer vision since the expression-toexpression translation is a non-linear and non-aligned problem. Moreover, it requires a high-level semantic understanding between the input and output images since the objects in images can have arbitrary poses, sizes, locations, backgrounds, and self-occlusions.To tackle this problem, we propose utilizing facial landmark information explicitly. Since it is a challenging problem, we split it into two sub-tasks, (i) category-guided landmark generation, and (ii) landmark-guided expression-to-expression translation. Two sub-tasks are trained in an end-to-end fashion that aims to enjoy the mutually improved benefits from the generated landmarks and expressions. Compared with current keypoint-guided approaches, the proposed LandmarkGAN only needs a single facial image to generate various expressions. Extensive experimental results on four public datasets demonstrate that the proposed LandmarkGAN achieves better results compared with state-of-the-art approaches only using a single image. The code is available at https://github.com/Ha0Tang/LandmarkGAN.

show abstract

“…Further, they can be used to generate novel examples not found in the original dataset [ 21 ]. Such feature learning also supports a variety of other applications, such as super-resolution [ 22 ], multimodal application [ 23 , 24 , 25 , 26 , 27 ], medical imaging [ 28 , 29 ], video prediction [ 30 , 31 , 32 , 33 , 34 ], natural language processing [ 35 , 36 , 37 ], transfer learning and zero-shot learning [ 38 ].…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Learning of Disentangled Representation via Auto-Encoding: A Survey

Eddahmani¹,

Pham

Napoléon³

et al. 2023

Sensors

View full text Add to dashboard Cite

In recent years, the rapid development of deep learning approaches has paved the way to explore the underlying factors that explain the data. In particular, several methods have been proposed to learn to identify and disentangle these underlying explanatory factors in order to improve the learning process and model generalization. However, extracting this representation with little or no supervision remains a key challenge in machine learning. In this paper, we provide a theoretical outlook on recent advances in the field of unsupervised representation learning with a focus on auto-encoding-based approaches and on the most well-known supervised disentanglement metrics. We cover the current state-of-the-art methods for learning disentangled representation in an unsupervised manner while pointing out the connection between each method and its added value on disentanglement. Further, we discuss how to quantify disentanglement and present an in-depth analysis of associated metrics. We conclude by carrying out a comparative evaluation of these metrics according to three criteria, (i) modularity, (ii) compactness and (iii) informativeness. Finally, we show that only the Mutual Information Gap score (MIG) meets all three criteria.

show abstract

Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model

Abstract: StyleCLIP with bangs double chin black hair with wrinkles pale* The work was done during Zipeng Xu's internship at VIS, Baidu.

Cited by 29 publications

References 30 publications

Disentangled Representation Learning for Recommendation

Disentangled Representation Learning for Recommendation

Facial Expression Translation Using Landmark Guided GANs

Unsupervised Learning of Disentangled Representation via Auto-Encoding: A Survey

Contact Info

Product

Resources

About