Image Search With Text Feedback by Visiolinguistic Attention Learning

Abstract

Image search with text feedback has promising impacts in various real-world applications, such as e-commerce and internet search. Given a reference image and text feedback from user, the goal is to retrieve images that not only resemble the input image, but also change certain aspects in accordance with the given text. This is a challenging task as it requires the synergistic understanding of both image and text. In this work, we tackle this task by a novel Visiolinguistic Attention Learning (VAL) framework. Specifically, we propose a composite transformer that can be seamlessly plugged in a CNN to selectively preserve and transform the visual features conditioned on language semantics. By inserting multiple composite transformers at varying depths, VAL is incentive to encapsulate the multi-granular visiolinguistic information, thus yielding an expressive representation for effective image search. We conduct comprehensive evaluation on three datasets: Fashion200k, Shoes and FashionIQ. Extensive experiments show our model exceeds existing approaches on all datasets, demonstrating consistent superiority in coping with various text feedbacks, including attribute-like and natural language descriptions.

Multi-Order Adversarial Representation Learning for Composed Query Image Retrieval

“…In order to verify the effectiveness of proposed methods, we compare state-of-the-art methods [8,26,7,27,23,9] on three datasets and cite performance scores from the original papers. However, we notice that CNN backbone used by these methods vary, so we train our models with ResNet18 or MobileNet for a fair comparison.…”

“…In [8], Vo et al learn a gating connection to retain the query image feature, and additionally employ a residual connection to fuse images and text. More recently, Chen et al [9] Fig. 1.…”

“…The main challenge for this task is how to learn a composite representation that captures both visual and semantic information. Previous studies have provided some approaches [7,8,9,10]. For instance, Santoro et al [7] first concatenate text features and image features simply, and a multilayer perceptron is further employed to fuse the cross-modal data.…”

“…However, the above works have not fully utilized the multi-level feature. Inspired by [9] that explores multiple multi-level features from convolutional neural networks, we go further in this direction by jointly exploring loworder features and high-order features on the basis of multi-level features. Besides, adversarial learning [11] has been proved to be effective in image generation (e.g., image-to-image translation [12,13], text-to-image generation [14]), as well as facilitating the fusion of image and text features [15].…”

Hierarchical Similarity Learning for Language-Based Product Image Retrieval

“…Cross-modal retrieval is a classical task at the intersection between computer vision and natural language processing, and has been widely explored [1,2,3,4]. Recently, with the increasing popularity of e-commerce platforms [5,6,7,8], language-based product image retrieval attracts increasing attention [9,10,11]. As exemplified in Fig.…”