Online multimodal deep similarity learning with application to image retrieval

Wu, Pengcheng; Hoi, Steven C. H.; Xia, Hao; Zhao, Peilin; Wang, Dayong; Miao, Chunyan

doi:10.1145/2502081.2502112

Cited by 152 publications

(70 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In doing so, we presented a simple and easy-to-implement architecture for acquiring semantic representations of both types of concept from linguistic and perceptual input. While neuro-probabilistic language models have been applied to the problem of multi-modal representation learning previously (Srivastava and Salakhutdinov, 2012;Wu et al, 2013;Silberer and Lapata, 2014) our model and experiments develop this work in several important ways. First, we address the problem of learning abstract concepts.…”

Section: Discussionmentioning

confidence: 99%

Learning Abstract Concept Embeddings from Multi-Modal Data: Since You Probably Can't See What I Mean

Hill

Korhonen

2014

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Models that acquire semantic representations from both linguistic and perceptual input are of interest to researchers in NLP because of the obvious parallels with human language learning. Performance advantages of the multi-modal approach over language-only models have been clearly established when models are required to learn concrete noun concepts. However, such concepts are comparatively rare in everyday language. In this work, we present a new means of extending the scope of multi-modal models to more commonly-occurring abstract lexical concepts via an approach that learns multimodal embeddings. Our architecture outperforms previous approaches in combining input from distinct modalities, and propagates perceptual information on concrete concepts to abstract concepts more effectively than alternatives. We discuss the implications of our results both for optimizing the performance of multi-modal models and for theories of abstract conceptual representation.

show abstract

Section: Discussionmentioning

confidence: 99%

Learning Abstract Concept Embeddings from Multi-Modal Data: Since You Probably Can't See What I Mean

Hill

Korhonen

2014

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…A fusion representation is learned simultaneously and used for the final prediction. Wu et al [17] use mutlimodal deep neural networks to learn a combined non-linear similarity function. They trained multiple deep denoising autoencoders for different low-level features in an unsupervised manner.…”

Section: Feature Transformation With Deep Neural Networkmentioning

confidence: 99%

“…Moreover, deep learning based multi-modal fusion methods have also been proposed recently [14], [15], [16]. For example, in [17], the distance metric between different modalities is learned by deep neural networks. The methods mentioned above are focused on the problem of how to utilize multiple features more effectively.…”

Section: Introductionmentioning

confidence: 99%

Heterogeneous Feature Selection With Multi-Modal Deep Neural Networks and Sparse Group LASSO

Zhao

Wang

2015

IEEE Trans. Multimedia

112

View full text Add to dashboard Cite

Abstract-Heterogeneous feature representations are widely used in machine learning and pattern recognition, especially for multimedia analysis. The multi-modal, often also highdimensional, features may contain redundant and irrelevant information that can deteriorate the performance of modeling in classification. It is a challenging problem to select the informative features for a given task from the redundant and heterogeneous feature groups. In this paper, we propose a novel framework to address this problem. This framework is composed of two modules, namely, multi-modal deep neural networks and feature selection with sparse group LASSO. Given diverse groups of discriminative features, the proposed technique first converts the multi-modal data into a unified representation with different branches of the multi-modal deep neural networks. Then, through solving a sparse group LASSO problem, the feature selection component is used to derive a weight vector to indicate the importance of the feature groups. Finally, the feature groups with large weights are considered more relevant and hence are selected. We evaluate our framework on three image classification datasets. Experimental results show that the proposed approach is effective in selecting the relevant feature groups and achieves competitive classification performance as compared with several recent baseline methods.

show abstract

“…Previous work has focused on projecting words and images onto a common space using a variety of methods including deep and restricted Boltzman machines [43], autoencoders [44], and recursive neural networks [45]. Similar methods were employed to combine other modalities such as speech and video or images ( [24], [43]).…”

Section: Multimodal Deep Learningmentioning

confidence: 99%

Visually Grounded Meaning Representations

Silberer

Ferrari

Lapata

2017

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Abstract-In this paper we address the problem of grounding distributional representations of lexical meaning. We introduce a new model which uses stacked autoencoders to learn higher-level representations from textual and visual input. The visual modality is encoded via vectors of attributes obtained automatically from images. We create a new large-scale taxonomy of 600 visual attributes representing more than 500 concepts and 700K images. We use this dataset to train attribute classifiers and integrate their predictions with text-based distributional models of word meaning. We evaluate our model on its ability to simulate word similarity judgments and concept categorization. On both tasks, our model yields a better fit to behavioral data compared to baselines and related models which either rely on a single modality or do not make use of attribute-based input.

show abstract

Online multimodal deep similarity learning with application to image retrieval

Cited by 152 publications

References 35 publications

Learning Abstract Concept Embeddings from Multi-Modal Data: Since You Probably Can't See What I Mean

Learning Abstract Concept Embeddings from Multi-Modal Data: Since You Probably Can't See What I Mean

Heterogeneous Feature Selection With Multi-Modal Deep Neural Networks and Sparse Group LASSO

Visually Grounded Meaning Representations

Contact Info

Product

Resources

About