Multimodal Convolutional Neural Networks for Matching Image and Sentence

Ma, Lin; Lu, Zhengdong; Shang, Lifeng; Li, Hang

doi:10.1109/iccv.2015.301

Cited by 304 publications

(186 citation statements)

References 42 publications

Supporting

Mentioning

185

Contrasting

Order By: Relevance

“…For NN approaches to contentbased image retrieval, see Wan et al (2014). Similarly, we exclude work on NN approaches to acoustic or multi-modal IR, such as mixing text with imagery (see Ma et al 2015bMa et al , 2016. We also intentionally focus on the current ''third wave'' revival of NN research, excluding earlier work.…”

mentioning

confidence: 99%

Neural information retrieval: at the end of the early years

et al. 2017

View full text Add to dashboard Cite

A recent ''third wave'' of neural network (NN) approaches now delivers state-ofthe-art performance in many machine learning tasks, spanning speech recognition, computer vision, and natural language processing. Because these modern NNs often comprise multiple interconnected layers, work in this area is often referred to as deep learning. Recent years have witnessed an explosive growth of research into NN-based approaches to information retrieval (IR). A significant body of work has now been created. In this paper,

show abstract

mentioning

confidence: 99%

Neural information retrieval: at the end of the early years

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Also we plan to investigate more complicated and flexible harmonization functions to adapt to the complex content of multimodal data. Further, we will apply the proposed method to more complicated multimodal correlation learning tasks, e.g., image-text matching [67]. We will study an extension of our method to be able to deal with missing data, since real-world multimodal data usually are unpaired or incomplete.…”

Section: Resultsmentioning

confidence: 99%

Harmonized Multimodal Learning with Gaussian Process Latent Variable Models

Song

Wang

Huang

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Multimodal learning aims to discover the relationship between multiple modalities. It has become an important research topic due to extensive multimodal applications such as cross-modal retrieval. This paper attempts to address the modality heterogeneity problem based on Gaussian process latent variable models (GPLVMs) to represent multimodal data in a common space. Previous multimodal GPLVM extensions generally adopt individual learning schemes on latent representations and kernel hyperparameters, which ignore their intrinsic relationship. To exploit strong complementarity among different modalities and GPLVM components, we develop a novel learning scheme called Harmonization, where latent model parameters are jointly learned from each other. Beyond the correlation fitting or intra-modal structure preservation paradigms widely used in existing studies, the harmonization is derived in a model-driven manner to encourage the agreement between modality-specific GP kernels and the similarity of latent representations. We present a range of multimodal learning models by incorporating the harmonization mechanism into several representative GPLVM-based approaches. Experimental results on four benchmark datasets show that the proposed models outperform the strong baselines for cross-modal retrieval tasks, and that the harmonized multimodal learning method is superior in discovering semantically consistent latent representation.

show abstract

“…Another similar method is Latent Dirichlet Allocation (LDA) [5,45] which attempts to model the correlation between multi-modal data. In recent years, deep architectures have also been conducted to learn the multimodal representation [43,8]. These models map the image representation and sentence representation into a common multi-modal embedding space.…”

Section: Related Workmentioning

confidence: 99%

Multi-modal gated recurrent units for image description

Yuan

2018

Multimed Tools Appl

View full text Add to dashboard Cite

Using a natural language sentence to describe the content of an image is a challenging but very important task. It is challenging because a description must not only capture objects contained in the image and the relationships among them, but also be relevant and grammatically correct. In this paper a multi-modal embedding model based on gated recurrent units (GRU) which can generate variable-length description for a given image. In the training step, we apply the convolutional neural network (CNN) to extract the image feature. Then the feature is imported into the multi-modal GRU as well as the corresponding sentence representations. The multi-modal GRU learns the inter-modal relations between image and sentence. And in the testing step, when an image is imported to our multi-modal GRU model, a sentence which describes the image content is generated. The experimental results demonstrate that our multi-modal GRU model obtains the state-of-the-art performance on Flickr8K, Flickr30K and MS COCO datasets.

show abstract

Multimodal Convolutional Neural Networks for Matching Image and Sentence

Cited by 304 publications

References 42 publications

Neural information retrieval: at the end of the early years

Neural information retrieval: at the end of the early years

Harmonized Multimodal Learning with Gaussian Process Latent Variable Models

Multi-modal gated recurrent units for image description

Contact Info

Product

Resources

About