Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking

Abstract

A major challenge in matching images and text is that they have intrinsically different data distributions and feature representations. Most existing approaches are based either on embedding or classification, the first one mapping image and text instances into a common embedding space for distance measuring, and the second one regarding image-text matching as a binary classification problem. Neither of these approaches can, however, balance the matching accuracy and model complexity well. We propose a novel framework that achieves remarkable matching performance with acceptable model complexity. Specifically, in the training stage, we propose a novel Multi-modal Tensor Fusion Network (MTFN) to explicitly learn an accurate image-text similarity function with rank-based tensor fusion rather than seeking a common embedding space for each image-text instance. Then, during testing, we deploy a generic Cross-modal Re-ranking (RR) scheme for refinement without requiring additional training procedure. Extensive experiments on two datasets demonstrate that our MTFN-RR consistently achieves the state-of-the-art matching performance with much less time complexity. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval. Figure 1: Illustration of the embedding-based (top) and the classification-based methods (bottom) that are typically used for image-text matching.

Target-Oriented Deformation of Visual-Semantic Embedding Space

“…Several recent methods for image-caption retrieval employ an object detector and a cross-modal attention. For example, SCO [16], SCAN [11], and MTFN [49] crop 10, 24, 36 regions, respectively, and merge them nonlinearly by using a cross-model attention network as introduced in Sect. 2.3.…”

“…† Note that MTFN[49] also proposed a re-ranking algorithm, which finds the best match between a set of queries and a set of targets (not between a single query and a set of targets). We omit this result because the problem setting is completely different from those of the other studies.…”

Rgb2lidar

“…Joint Embedding: Joint embedding models have shown excellent performance on several multimedia tasks, e.g., cross-modal retrieval [10,19,31,38,53,54,75,77,82,84], image captioning [35,49], image classification [23,25,32] video summarization [15,58], crossview matching [81]. Cross-modal retrieval methods require computing similarity between two different modalities, e.g., RGB and depth.…”

SGD_Tucker: A Novel Stochastic Optimization Strategy for Scalable Parallel Sparse Tucker Decomposition

“…T ENSORS are a widely used data representation style for interaction data in the Machine Learning (ML) application community [1], e.g, in Recommendation Systems [2], Quality of Service (QoS) [3], Network Flow [4], Cyber-Physical-Social (CPS) [5], or Social Networks [6]. In addition to applications in which the data is naturally represented in the form of tensors, another common used case is the fusion in multi-view or multi-modality problems [7]. Here, during the learning process, each modality corresponds to a feature and the feature alignment involves fusion.…”

“…Here, during the learning process, each modality corresponds to a feature and the feature alignment involves fusion. Tensors are a common form of feature fusion for multi-modal learning [7], [8], [9], [10]. Unfortunately, tensors can be difficult to process in practice.…”