Generalized Semantic Preserving Hashing for N-Label Cross-Modal Retrieval

Mandal, D. K.; Chaudhury, Kunal N.; Biswas, Soma

doi:10.1109/cvpr.2017.282

Cited by 114 publications

(67 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…I→T I→A I→V T→I T→A T→V A→I A→T A→V V→I V→T V→A Average Our FGCrossNet 0.210 0.526 0.606 0.255 0.181 0.208 0.553 0.159 0.443 0.629 0.195 0.437 0.366 MHTN [20] 0.116 0.195 0.281 0.124 0.138 0.185 0.196 0.127 0.290 0.306 0.186 0.306 0.204 ACMR [21] 0.162 0.119 0.477 0.075 0.015 0.081 0.128 0.028 0.068 0.536 0.138 0.111 0.162 JRL [22] 0.160 0.085 0.435 0.190 0.028 0.095 0.115 0.035 0.065 0.517 0.126 0.068 0.160 GSPH [23] 0.140 0.098 0.413 0.179 0.024 0.109 0.129 0.024 0.073 0.512 0.126 0.086 0.159 CMDN [24] 0.099 0.009 0.377 0.123 0.007 0.078 0.017 0.008 0.010 0.446 0.081 0.009 0.105 SCAN [25] 0.050…”

Section: Methodsunclassified

See 1 more Smart Citation

A New Benchmark and Approach for Fine-grained Cross-media Retrieval

Peng

Xie

2019

Proceedings of the 27th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Cross-media retrieval is to return the results of various media types corresponding to the query of any media type. Existing researches generally focus on coarse-grained cross-media retrieval. When users submit an image of "Slaty-backed Gull" as a query, coarsegrained cross-media retrieval treats it as "Bird", so that users can only get the results of "Bird", which may include other bird species with similar appearance (image and video), descriptions (text) or sounds (audio), such as "Herring Gull". Such coarse-grained crossmedia retrieval is not consistent with human lifestyle, where we generally have the fine-grained requirement of returning the exactly relevant results of "Slaty-backed Gull" instead of "Herring Gull". However, few researches focus on fine-grained cross-media retrieval, which is a highly challenging and practical task. Therefore, in this paper, we first construct a new benchmark for fine-grained cross-media retrieval, which consists of 200 fine-grained subcategories of the "Bird", and contains 4 media types, including image, text, video and audio. To the best of our knowledge, it is the first benchmark with 4 media types for fine-grained cross-media retrieval. Then, we propose a uniform deep model, namely FGCross-Net, which simultaneously learns 4 types of media without discriminative treatments. We jointly consider three constraints for better common representation learning: classification constraint ensures the learning of discriminative features for fine-grained subcategories, center constraint ensures the compactness characteristic of the features of the same subcategory, and ranking constraint ensures the sparsity characteristic of the features of different subcategories. Extensive experiments verify the usefulness of the new benchmark and the effectiveness of our FGCrossNet. The new benchmark and the source code of FGCrossNet will be made available at https://github.com/PKU-ICST-MIPL/FGCrossNet_ACMMM2019. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval; • Computing methodologies → Artificial intelligence.

show abstract

Section: Methodsunclassified

“…We compare our FGCrossNet with state-of-the-art cross-media retrieval methods, including MHTN [20], ACMR [21], JRL [22], GSPH [23], CMDN [24], SCAN [25], GXN [26]. MHTN [20] learns common representations for 5 media types by transferring knowledge from single-media source domain (image) to cross-media target domain.…”

Section: Compared Methodsmentioning

confidence: 99%

A New Benchmark and Approach for Fine-grained Cross-media Retrieval

Peng

Xie

2019

Proceedings of the 27th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…More specifically, only one of the c entries is equal to 1 if the data is annotated with single label (e.g., L i x = [0 0 1 0 0]), and more than one entries will be equal to 1 if this data is marked with multiple labels (e.g., L j y = [1 0 1 0 1]). As suggested in [7], [24], semantic affinity matrix with embedding supervision can be efficiently utilized to learn hash codes of training instances. Accordingly, we first construct an affinity matrix…”

Section: Problem Formulationmentioning

confidence: 99%

“…Nevertheless, such unified hash code could inherently sacrifice their representation capability and scalability because it cannot guarantee the learned binary codes to be semantically discriminative for heterogeneous data representation. In addition, the majority of existing cross-modal hashing approaches mainly arXiv:1805.01963v1 [cs.CV] 4 May 2018 focus on the paired multi-modal collections, and very few works, except for [7], have been designed to handle the unpaired scenarios. Remarkably, all these approaches select the equalized hash length to characterize the multi-modal data and make them directly comparable in an isomorphic Hamming space.…”

Section: Introductionmentioning

confidence: 99%

MTFH: A Matrix Tri-Factorization Hashing Framework for Efficient Cross-Modal Retrieval

Liu

Ling

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

152

View full text Add to dashboard Cite

Hashing has recently sparked a great revolution in cross-modal retrieval due to its low storage cost and high query speed. Most existing cross-modal hashing methods learn unified hash codes in a common Hamming space to represent all multi-modal data and make them intuitively comparable. However, such unified hash codes could inherently sacrifice their representation scalability because the data from different modalities may not have one-to-one correspondence and could be stored more efficiently by different hash codes of unequal lengths. To mitigate this problem, this paper proposes a generalized and flexible cross-modal hashing framework, termed Matrix Tri-Factorization Hashing (MTFH), which not only preserves the semantic similarity between the multi-modal data points, but also works seamlessly in various settings including paired or unpaired multi-modal data, and equal or varying hash length encoding scenarios. Specifically, MTFH exploits an efficient objective function to jointly learn the flexible modality-specific hash codes with different length settings, while simultaneously excavating two semantic correlation matrices to ensure heterogeneous data comparable. As a result, the derived hash codes are more semantically meaningful for various challenging cross-modal retrieval tasks. Extensive experiments evaluated on public benchmark datasets highlight the superiority of MTFH under various retrieval scenarios and show its very competitive performance with the state-of-the-arts.

show abstract

“…After studying the base case, we consider the effect of the latent loss (L 2 ) by examining the performance of the full model, excluding the L 2 loss in Equation 6. It is found that L 2 boosts the performance of the model quite meagerly.…”

Section: Critical Analysismentioning

confidence: 99%

CMIR-NET : A deep learning based model for cross-modal retrieval in remote sensing

Chaudhuri

Banerjee

Bhattacharya

et al. 2020

Pattern Recognition Letters

View full text Add to dashboard Cite

We address the problem of cross-modal information retrieval in the domain of remote sensing. In particular, we are interested in two application scenarios: i) crossmodal retrieval between panchromatic (PAN) and multi-spectral imagery, and ii) multi-label image retrieval between very high resolution (VHR) images and speech based label annotations. These multi-modal retrieval scenarios are more challenging than the traditional uni-modal retrieval approaches given the inherent differences in distributions between the modalities. However, with the increasing availability of multi-source remote sensing data and the scarcity of enough semantic annotations, the task of multi-modal retrieval has recently become extremely important. In this regard, we propose a novel deep neural network based architecture which is considered to learn a discriminative shared feature space for all the input modalities, suitable for semantically coherent information retrieval. Extensive experiments are carried out on the benchmark large-scale PAN -multi-spectral DSRSID dataset and the multi-label UC-Merced dataset. Together with the Merced dataset, we generate a corpus of speech signals corresponding to the labels. Superior performance with respect to the current state-of-the-art is observed in all the cases. IntroductionWith the availability of extensive data collections, and with efficient storage facilities, the recent time has witnessed the accumulation of an enormous volume of remote sensing data. These data represent different modalities regarding the same underlying phenomenon. This has, in turn, led to the requirement for shifting the paradigm from the traditional uni-modal to the more challenging cross-modal information extraction scenario. Ideally, given a query from one domain, the cross-modal approaches retrieve relevant information from the other domains. In this regard, the necessity of the multi-modal approaches is especially more evident in the area of remote sensing (RS) data analysis, due to the availability of a large number of satellite missions. As an example, given a panchromatic query image, we may want to retrieve all semantically consistent multi-spectral images captured by a different satellite sensor for better analysis in the spectral domain. In gist, the paradigm of cross-modal information retrieval is critical in the area of RS given the complementary information captured in different cross-sensor acquisitions.Although the notion of image retrieval has received extensive attention, several other cross-modal combinations are tried in the past, specifically in the area of computer vision and natural language processing: image -text (label) pairs [1,2], RBG image -depth image pairs [3], etc. While most of the recent endeavors in this respect follow a closed-form formulation to associate the samples from different domains directly [1,4], others [5, 2, 6] depend on the conventional learning-based approaches. In contrast to the single-label based image retrieval, few strategies extend the framework to support multiple...

show abstract

Generalized Semantic Preserving Hashing for N-Label Cross-Modal Retrieval

Cited by 114 publications

References 19 publications

A New Benchmark and Approach for Fine-grained Cross-media Retrieval

A New Benchmark and Approach for Fine-grained Cross-media Retrieval

MTFH: A Matrix Tri-Factorization Hashing Framework for Efficient Cross-Modal Retrieval

CMIR-NET : A deep learning based model for cross-modal retrieval in remote sensing

Contact Info

Product

Resources

About