Semi-Supervised Cross-Media Feature Learning With Unified Patch Graph Regularization

Abstract

With the rapid growth of multimedia data such as text, image, video, audio and 3D model, cross-media retrieval has become increasingly important, because users can retrieve the results with various types of media by submitting a query of any media type. Comparing with single-media retrieval such as image retrieval and text retrieval, cross-media retrieval is better because it provides the retrieval results with all kinds of media at the same time. In this paper, we focus on how to learn cross-media features for different media types, which is a key challenge for cross-media retrieval. Existing methods either model different media types separately or only exploit the labeled multimedia data. Actually, data from different media types with the same semantic category are complementary to each other, and jointly modeling them is able to improve the accuracy of cross-media retrieval. In addition, although the labeled data are accurate, they require a lot of human labor and thus are very scarce. To address the above problems, we propose a semi-supervised crossmedia feature learning algorithm with unified patch graph regularization (S 2 UPG). Our motivation and contributions mainly lie in the following three aspects: (1) Existing methods only model different media types in different graphs, while we employ one joint graph to simultaneously model all the media types. The joint graph is able to fully exploit the semantic correlations among various media types, which are complementary to provide the rich hint for cross-media correlation. (2) Existing methods only consider the original media instances (such as images, videos, texts, audios, and 3D models) but ignore their patches, while we make full use of both the media instances and their patches in one graph. Cross-media patches could emphasize the important parts and make cross-media correlations more precise.(3) Traditional semi-supervised learning methods only exploit single-media unlabeled instances, while our approach fully exploits cross-media unlabeled instances and their patches, which can increase the diversity of training data and boost the accuracy of cross-media retrieval. Comparing with the current state-of-theart methods on 3 datasets, including the challenging XMedia dataset with 5 media types, the comprehensive experimental results show that our proposed approach performs better.

Modality-Specific Cross-Modal Similarity Measurement With Recurrent Attention Network

“…Furthermore, Joint Representation Learning (JRL) [9] is proposed to construct several separate graphs for different modalities and learn projection matrices with the joint consideration of correlation and semantic information. Peng et al [24] further improve the previous works [9], [23] by constructing a unified hypergraph to learn the common space for up to five modalities, which also utilizes the fine-grained information at the same time. Besides, Wang et al [10] also adopt multimodal graph regularization term to preserve inter-modality and intra-modality similarity relationships.…”

Semi-Supervised Cross-Modal Retrieval With Label Prediction

“…Semi-supervised methods Zhai et al [2014] Peng et al [2016] Zhang et al [2018 try to bridge the gap between the supervised and unsupervised methods by utilizing both labeled as well as unlabeled data for training, and it is much less explored compared to the others. Zhai et al [2014] establishes an unified optimization framework to jointly associate the relationship between the correlation and the semantic information of the training examples.…”

Cross-media analysis and reasoning: advances and directions

“…To overcome this limitation, researchers have also attempted to develop datasets and methods for scenarios with more media types. For example, the newly constructed XMedia dataset (http://www.icst.pku.edu.cn/mipl/XMedia) is the first dataset containing five media types (text, image, video, audio, and 3D model), and methods such as those proposed by Zhai et al (2014) and Peng et al (2016b) can jointly model the correlations and semantic information in a unified framework with graph regularization for the five media types on the XMedia dataset. Yang et al (2008) introduced another model called the multimedia document (MMD) to represent data, where each MMD is a set of media objects of different modalities but carrying the same semantics.…”

“…As discussed in Pan (2016), cross-media intelligence plays the role of a cornerstone in artificial intelligence, through which the machines can recognize the external environment. Although considerable improvement has been made in the research of crossmedia analysis and reasoning (Rasiwasia et al, 2010;Yang et al, 2012;Peng et al, 2016a;2016b), there remain some important challenges and unclear points in future research directions. In this paper, we give a comprehensive overview of not only the advances achieved by existing studies, but also future directions for cross-media analysis and reasoning.…”