EI-CLIP: Entity-aware Interventional Contrastive Learning for E-commerce Cross-modal Retrieval

Ma, Haoyu; Zhao, Handong; Lin, Zhe; Kale, Ajinkya; Wang, Zhangyang; Yu, Tong; Gu, Jiuxiang; Choudhary, Sunav; Xie, Xiaohui

doi:10.1109/cvpr52688.2022.01752

Cited by 37 publications

(16 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Radford et al [8] also provided a CNN version of the image encoder. However, due to the advantage of ViT on pre-training, the ViT encoder outperformed the CNN encoder, and it is most commonly applied in other works [9]. CLIP has recently been used in various tasks, such as e-commerce image retrieval [9], text-image generation [10], and image segmentation [18].…”

Section: The Network Of Clipmentioning

confidence: 99%

“…However, due to the advantage of ViT on pre-training, the ViT encoder outperformed the CNN encoder, and it is most commonly applied in other works [9]. CLIP has recently been used in various tasks, such as e-commerce image retrieval [9], text-image generation [10], and image segmentation [18]. However, CLIP still lacks the ability to effectively match local information in images to their descriptions in cross-modal information retrieval tasks [12].…”

Section: The Network Of Clipmentioning

confidence: 99%

“…Recently, Radford et al [8] proposed a Contrastive Language-Image Pre-training network (CLIP) which achieved state-of-the-art performance in cross-modal tasks. CLIP employs the pre-trained GPT-2 and Visual Transformer (ViT) to encode descriptions and images into a shared embedding space respectively [8], and has been widely applied to various information retrieval tasks such as e-commerce image retrieval [9], and text-image generation [10]. One of CLIP's limitations is that it cannot identify relations between objects in images.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Clip-Rr: Improved Clip Network for Relation-Focused Cross-Modal Information Retrieval

Yan

Cosma

2023

Preprint

View full text Add to dashboard Cite

Section: The Network Of Clipmentioning

confidence: 99%

Section: The Network Of Clipmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Clip-Rr: Improved Clip Network for Relation-Focused Cross-Modal Information Retrieval

Yan

Cosma

2023

Preprint

View full text Add to dashboard Cite

“…The problem of semantic misunderstanding has also been investigated by previous works. EI-CLIP [33] considers the problem of cross-modal retrieval in the field of E-commerce. Sharing similar insight with our work, the authors notice the model bias towards some specific word tokens in CLIP, and introduce causal inference to align the text encoder with e-commerce domain knowledge.…”

Section: Related Workmentioning

confidence: 99%

Contrastive Language-Image Pre-Training with Knowledge Graphs

Pan¹,

Tian-zhu²,

Han³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model [38]. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.

show abstract

“…In our case, we are facing two challenges in building the fashion-domain MTL model: (1) Architecturally, it is non-trivial to model the diverse tasks in one unified architecture. Taking the popular CLIP [60] as an example, its two-stream architecture is designed for image-text alignment [52] and thus lacks the modality fusion mechanism as required by many V+L fashion tasks (e.g., text-guided image retrieval [2,83] and image captioning [85]). (2) In terms of optimization, a fashion-domain MTL model is prone to the notorious negative transfer problem [8,13,36,46,56,63] due to both task input/output format differences and imbalanced dataset sizes.…”

Section: Introductionmentioning

confidence: 99%

FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks

Han¹,

Zhu²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

In the fashion domain, there exists a variety of visionand-language (V+L) tasks, including cross-modal retrieval, text-guided image retrieval, multi-modal classification, and image captioning. They differ drastically in each individual input/output format and dataset size. It has been common to design a task-specific model and fine-tune it independently from a pre-trained V+L model (e.g., CLIP). This results in parameter inefficiency and inability to exploit inter-task relatedness. To address such issues, we propose a novel FAshion-focused Multi-task Efficient learning method for Vision-and-Language tasks (FAME-ViL) in this work. Compared with existing approaches, FAME-ViL applies a single model for multiple heterogeneous fashion tasks, therefore being much more parameter-efficient. It is enabled by two novel components: (1) a task-versatile architecture with cross-attention adapters and task-specific adapters integrated into a unified V+L model, and (2) a stable and effective multi-task training strategy that supports learning from heterogeneous data and prevents negative transfer. Extensive experiments on four fashion tasks show that our FAME-ViL can save 61.5% of parameters over alternatives, while significantly outperforming the conventional independently trained single-task models. Code is available at https://github.com/BrandonHanx/FAME-ViL.

show abstract

EI-CLIP: Entity-aware Interventional Contrastive Learning for E-commerce Cross-modal Retrieval

Cited by 37 publications

References 37 publications

Clip-Rr: Improved Clip Network for Relation-Focused Cross-Modal Information Retrieval

Clip-Rr: Improved Clip Network for Relation-Focused Cross-Modal Information Retrieval

Contrastive Language-Image Pre-Training with Knowledge Graphs

FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks

Contact Info

Product

Resources

About