Dual Compositional Learning in Interactive Image Retrieval

Abstract

We present an approach named Dual Composition Network (DCNet) for interactive image retrieval that searches for the best target image for a natural language query and a reference image. To accomplish this task, existing methods have focused on learning a composite representation of the reference image and the text query to be as close to the embedding of the target image as possible. We refer this approach as Composition Network. In this work, we propose to close the loop with Correction Network that models the difference between the reference and target image in the embedding space and matches it with the embedding of the text query. That is, we consider two cyclic directional mappings for triplets of (reference image, text query, target image) by using both Composition Network and Correction Network. We also propose a joint training loss that can further improve the robustness of multimodal representation learning. We evaluate the proposed model on three benchmark datasets for multimodal retrieval: Fashion-IQ, Shoes, and Fashion200K. Our experiments show that our DCNet achieves new state-of-the-art performance on all three datasets, and the addition of Correction Network consistently improves multiple existing methods that are solely based on Composition Network. Moreover, an ensemble of our model won the first place in Fashion-IQ 2020 challenge held in a CVPR 2020 workshop.

Conditioned Image Retrieval for Fashion using Contrastive Learning and CLIP-based Features

“…Table 3 shows the quantitative results on Fashion-IQ validation set. Our approach outperforms the state-of-the-art by improving up to ∼ 5% in average R@10 and 3% in average R@50 upon the best method, DCNet [17], when using the CLIP RN50x4 backbone. Our method have the highest recall in the Shirt and Toptee categories, with comparable performance in the Dress category, using both backbones.…”

“…A schema of the complete system is shown in Figure 1 on page 1. In contrast to previous works like [6,17,19,24] that build from different image and textual model, we start from the hypothesis of having a common embedding of images and text, realized by CLIP. As shown in [22], similar concepts expressed in text and images tend to share similar features, or at least be "near" in the common space.…”

“…In [19], image style and content are considered separately by two different neural network modules. In [17] a Correction Network is added which explicitly models the difference between the reference and target image in the embedding space.…”

“…Instead, in [19,27] features extracted from the backbone are 3-dimensional and the composition takes care of spatial information, in [6] the features are extracted at different convolutional layers from the ResNet-50 backbone. In [17] the authors divided the image and the sentence into a set of localized components assigning a representation module, denoted as experts, to each of them. More similar to our work is [24] which trains a combiner directly on flattened image and text features that, differently from our work, are obtained from different embeddings.…”

“…Dress Toptee Average Method R@10 R@50 R@10 R@50 R@10 R@50 R@10 R@50 JVSM [5] 12 We follow experimental setting as in [17,19]. We employ the average recall at rank K (Recall@K) as evaluation metric, namely Recall@10 (R@10) and Recall@50 (R@50).…”