Heterogeneous Feature Fusion and Cross-modal Alignment for Composed Image Retrieval

Zhang, Gangjian; Wei, Shikui; Pang, Huaxin; Zhao, Yao

doi:10.1145/3474085.3475659

Cited by 22 publications

(4 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compared TG-CIR with the following baselines: TIRG [32], VAL [5], CIRPLANT [24], CosMo [21], DATIR [11], MCR [38], CLVC-Net [35], ARTEMIS [7], EER [37], FashionViL [16], CRR [36], AMC [41], Clip4cir [1], and FAME-ViL [16]. Notably, the former twelve baselines use traditional models like ResNet [18] and LSTM [20] as the feature extraction backbone, while the latter two take advantage of the multimodal pre-trained large model CLIP.…”

Section: Performance Comparisonmentioning

confidence: 99%

“…For multimodal query composition, existing methods devote to design various neural networks [1,3,5,7,12,13,21,32,35,38] to compose the multimodal query, but overlook to model the intrinsic conflicting relationship between the multimodal query. Figure 1 illustrates an example of multimodal query, where the reference image indicates that the user may want a white princess dress, while the modification text specifies that the user wants to change the color and style of the reference image to "black" and "more elegant", respectively.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Target-Guided Composed Image Retrieval

Wen,

Zhang,

Song

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Composed image retrieval (CIR) is a new and flexible image retrieval paradigm, which can retrieve the target image for a multimodal query, including a reference image and its corresponding modification text. Although existing efforts have achieved compelling success, they overlook the conflict relationship modeling between the reference image and the modification text for improving the multimodal query composition and the adaptive matching degree modeling for promoting the ranking of the candidate images that could present different levels of matching degrees with the given query. To address these two limitations, in this work, we propose a Target-Guided Composed Image Retrieval network (TG-CIR). In particular, TG-CIR first extracts the unified global and local attribute features for the reference/target image and the modification text with the contrastive language-image pre-training model (CLIP) as the backbone, where an orthogonal regularization is introduced to promote the independence among the attribute features. Then TG-CIR designs a target-query relationship-guided multimodal query composition module, comprising a target-free student composition branch and a target-based teacher composition branch, where the target-query relationship is injected into the teacher branch for guiding the conflict relationship modeling of the student branch. Last, apart from the conventional batch-based classification loss, TG-CIR additionally introduces a batch-based target similarity-guided matching degree regularization to promote the metric learning process. Extensive experiments on three benchmark datasets demonstrate the superiority of our proposed method. CCS CONCEPTS• Information systems → Image search.

show abstract

Section: Performance Comparisonmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Target-Guided Composed Image Retrieval

Wen,

Zhang,

Song

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…intermodal attention (Wehrmann et al, 2020) and selfattention mechanisms (Han et al, 2021). Zhang et al (2021b) design a cross-modal guided pooling module that attends to local information dynamically. These sophisticated aggregators typically require more time, and don't always outperform simple pooling strategies.…”

Section: Related Workmentioning

confidence: 99%

Improving Visual-Semantic Embedding with Adaptive Pooling and Optimization Objective

Zhang,

Shu,

Xiao

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

Visual-Semantic Embedding (VSE) aims to learn an embedding space where related visual and semantic instances are close to each other. Recent VSE models tend to design complex structures to pool visual and semantic features into fixed-length vectors and use hard triplet loss for optimization. However, we find that: (1) combining simple pooling methods is no worse than these sophisticated methods; and (2) only considering the most difficult-to-distinguish negative sample leads to slow convergence and poor Recall@K improvement. To this end, we propose an adaptive pooling strategy that allows the model to learn how to aggregate features through a combination of simple pooling methods. We also introduce a strategy to dynamically select a group of negative samples to make the optimization converge faster and perform better. Experimental results on Flickr30K and MS-COCO demonstrate that a standard VSE using our pooling and optimization strategies outperforms current state-of-the-art systems (at least 1.0% on the metrics of recall) in imageto-text and text-to-image retrieval. Source code of our experiments is available at https: //github.com/96-Zachary/vse_2ad.

show abstract

“…A compositor plays a fundamental role to integrate the textual information with the imagery modality. TGR compositors have been proposed based on various techniques, such as gating mechanism [49], hierarchical attention [7,23,12,20], graph neural network [54,44], joint learning [6,27,44,52,55], ensemble learning [50], style-content modification [29,5] and vision & language pre-training [32].…”

Section: Related Workmentioning

confidence: 99%

UIGR: Unified Interactive Garment Retrieval

Han¹,

He²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Interactive garment retrieval (IGR) aims to retrieve a target garment image based on a reference garment image along with user feedback on what to change on the reference garment. Two IGR tasks have been studied extensively: text-guided garment retrieval (TGR) and visually compatible garment retrieval (VCR). The user feedback for the former indicates what semantic attributes to change with the garment category preserved, while the category is the only thing to be changed explicitly for the latter, with an implicit requirement on style preservation. Despite the similarity between these two tasks and the practical need for an efficient system tackling both, they have never been unified and modeled jointly. In this paper, we propose a Unified Interactive Garment Retrieval (UIGR) framework to unify TGR and VCR. To this end, we first contribute a large-scale benchmark suited for both problems. We further propose a strong baseline architecture to integrate TGR and VCR in one model. Extensive experiments suggest that unifying two tasks in one framework is not only more efficient by requiring a single model only, it also leads to better performance. Code and datasets are available at GitHub.

show abstract

Heterogeneous Feature Fusion and Cross-modal Alignment for Composed Image Retrieval

Cited by 22 publications

References 26 publications

Target-Guided Composed Image Retrieval

Target-Guided Composed Image Retrieval

Improving Visual-Semantic Embedding with Adaptive Pooling and Optimization Objective

UIGR: Unified Interactive Garment Retrieval

Contact Info

Product

Resources

About