Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval

Mithun, Niluthpol Chowdhury; Panda, Rameswar; Papalexakis, Evangelos E.; Roy-Chowdhury, Amit K.

doi:10.1145/3240508.3240712

Cited by 59 publications

(16 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, without extra information, improvement is limited. Chowdhury et al [44] introduced additional web information to cross-modal retrieval.…”

Section: Deep Learning Methodsmentioning

confidence: 99%

Abstraction and Association: Cross-Modal Retrieval Based on Consistency between Semantic Structures

Zheng

Ren

Liu

et al. 2020

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Cross-modal retrieval aims to find relevant data of different modalities, such as images and text. In order to bridge the modality gap, most existing methods require a lot of coupled sample pairs as training data. To reduce the demands for training data, we propose a cross-modal retrieval framework that utilizes both coupled and uncoupled samples. The framework consists of two parts: Abstraction that aims to provide high-level single-modal representations with uncoupled samples; then, Association links different modalities through a few coupled training samples. Moreover, under this framework, we implement a cross-modal retrieval method based on the consistency between the semantic structure of multiple modalities. First, both images and text are represented with the semantic structure-based representation, which represents each sample as its similarity from the reference points that are generated from single-modal clustering. Then, the reference points of different modalities are aligned through an active learning strategy. Finally, the cross-modal similarity can be measured with the consistency between the semantic structures. The experiment results demonstrate that given proper abstraction of single-modal data, the relationship between different modalities can be simplified, and even limited coupled cross-modal training data are sufficient for satisfactory retrieval accuracy.

show abstract

“…Nevertheless, without extra information, improvement is limited. Chowdhury et al [44] introduced additional web information to cross-modal retrieval.…”

Section: Deep Learning Methodsmentioning

confidence: 99%

Abstraction and Association: Cross-Modal Retrieval Based on Consistency between Semantic Structures

Zheng

Ren

Liu

et al. 2020

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…Numerous publications in recent years deal with multimodal information in retrieval tasks. The general problem of reduc-ing or bridging the semantic gap [44] between images and text is the main issue in cross-media retrieval [3,34,35,39,50]. Fan et al [8] tackle this problem by modeling humans' visual and descriptive senses with a multi-sensory fusion network.…”

Section: Multimedia Information Retrievalmentioning

confidence: 99%

Characterization and classification of semantic image-text relations

Otto

Springstein

Anand

et al. 2020

Int J Multimed Info Retr

View full text Add to dashboard Cite

The beneficial, complementary nature of visual and textual information to convey information is widely known, for example, in entertainment, news, advertisements, science, or education. While the complex interplay of image and text to form semantic meaning has been thoroughly studied in linguistics and communication sciences for several decades, computer vision and multimedia research remained on the surface of the problem more or less. An exception is previous work that introduced the two metrics Cross-Modal Mutual Information and Semantic Correlation in order to model complex image-text relations. In this paper, we motivate the necessity of an additional metric called Status in order to cover complex image-text relations more completely. This set of metrics enables us to derive a novel categorization of eight semantic image-text classes based on three dimensions. In addition, we demonstrate how to automatically gather and augment a dataset for these classes from the Web. Further, we present a deep learning system to automatically predict either of the three metrics, as well as a system to directly predict the eight image-text classes. Experimental results show the feasibility of the approach, whereby the predict-all approach outperforms the cascaded approach of the metric classifiers.

show abstract

“…CHAIN-VSE [16] builds a bidirectional retrieval framework which relies on character-level inception module for visualsemantic embeddings. Mithun et al [41] build a two-stage approach for image-text retrieval by using a supervised ranking loss combined with weakly-supervised web images to learn multimodal representation. However, these methods cannot well capture the fine-grained correlation between image and text.…”

Section: Related Workmentioning

confidence: 99%

Adversarial Attentive Multi-Modal Embedding Learning for Image-Text Matching

Wei

Zhou

2020

IEEE Access

View full text Add to dashboard Cite

Matching the image and text with deep models has been extensively studied in recent years. Mining the correlation between image and text to learn effective multi-modal features is crucial for image-text matching. However, most existing approaches model the different types of correlation independently. In this work, we propose a novel model named Adversarial Attentive Multi-modal Embedding Learning (AAMEL) for image-text matching. It combines adversarial networks and attention mechanism to learn effective and robust multi-modal embeddings for better matching between the image and text. Adversarial learning is implemented as an interplay between two processes. First, two attention models are proposed to exploit two types of correlation between the image and text for multi-modal embedding learning and to confuse the other process. Then the discriminator tries to distinguish the two types of multi-modal embeddings learned by the two attention models, in which the two attention models are reinforced mutually. Through adversarial learning, it is expected that both the two embeddings from the attention models can well exploit the two types of correlation, and thus they can deceive the discriminator that they are generated from the other attention-based model. By integrating the attention mechanism and adversarial learning, the learned multi-modal embeddings are more effective for image and text matching. Extensive experiments have been conducted on the benchmark datasets of Flickr30K and MSCOCO to demonstrate the superiority of the proposed approaches over the state-of-the-art methods on image-text retrieval.

show abstract

Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval

Cited by 59 publications

References 59 publications

Abstraction and Association: Cross-Modal Retrieval Based on Consistency between Semantic Structures

Abstraction and Association: Cross-Modal Retrieval Based on Consistency between Semantic Structures

Characterization and classification of semantic image-text relations

Adversarial Attentive Multi-Modal Embedding Learning for Image-Text Matching

Contact Info

Product

Resources

About