Deep Multi-Semantic Fusion-Based Cross-Modal Hashing

Abstract

Due to the low costs of its storage and search, the cross-modal retrieval hashing method has received much research interest in the big data era. Due to the application of deep learning, the cross-modal representation capabilities have risen markedly. However, the existing deep hashing methods cannot consider multi-label semantic learning and cross-modal similarity learning simultaneously. That means potential semantic correlations among multimedia data are not fully excavated from multi-category labels, which also affects the original similarity preserving of cross-modal hash codes. To this end, this paper proposes deep multi-semantic fusion-based cross-modal hashing (DMSFH), which uses two deep neural networks to extract cross-modal features, and uses a multi-label semantic fusion method to improve cross-modal consistent semantic discrimination learning. Moreover, a graph regularization method is combined with inter-modal and intra-modal pairwise loss to preserve the nearest neighbor relationship between data in Hamming subspace. Thus, DMSFH not only retains semantic similarity between multi-modal data, but integrates multi-label information into modal learning as well. Extensive experimental results on two commonly used benchmark datasets show that our DMSFH is competitive with the state-of-the-art methods.

A Cross-Modal Hash Retrieval Method with Fused Triples

“…In addition, there was also a Samsung 980 Pro2T solid-state drive. In this paper, eight advanced cross-modal retrieval methods were selected to compare with Tri-CMH, namely CMFH [19], CCA-ITQ [18], SCM [7], SePH [23], DCMH [15], TDH [27], DLFH [2], and DMSFH [25]. Among them, the first four algorithms were based on shallow frameworks, and the last four were based on deep learning.…”

“…The hash function can be learned in conjunction with semantic tagging information during feature extraction, thereby reducing semantic differences between modalities to improve the performance of cross-modal retrieval [20]. Typical supervised cross-modal hashing methods include the discrete latent factor hashing (DLFH) crossmodal method [2], the semantic correlation maximization (SCM) method [7], the deep cross-modal hashing (DCMH) method [15], the generalized semantic preserving hashing for n-label cross-modal retrieval [21], the multimodal latent binary embedding (MLBE) method [22], the semantic preservation hashing (SePH) method [23], the cross-view hashing (CVH) method [24], the deep multi-semantic fusion-based cross-modal hashing (DMSFH) method [25], the deep visual-semantic hashing (DVSH) method [26] and the triplet-based deep hashing (TDH) method [27]. [18] is a typical correlation analysis iterative quantification methodology proposed by Gong Y. et al in 2012. It is an unsupervised cross-modal hash retrieval method that provides multivariate statistics by evaluating the similarity of two sets of variables.…”

“…DMSFH [25] proposed in 2022 by Zhu X et al, consists of deep multi-semantic fusionbased cross-modal hashing, which uses a multi-label semantic fusion method to improve cross-modal consistent semantic discrimination learning. Moreover, a graph regularization method combines inter-modal and intra-modal pairwise loss to preserve the nearest neighbor relationship between data in the Hamming subspace.…”

Cross-modal retrieval based on multi-dimensional feature fusion hashing

“…Using cosine distance and Euclidean distance, the same measurement index can accurately reflect the similarity between different modal data in Deep Semantic Cross-Modal Hashing Based on Graph Similarity of Modal-Specific (DCMHGMS) [36]. The distance between similar data can be reduced by constructing ranking alignment loss to unearth the semantic structure between different modal data in Deep Rank Cross-modal Hashing (DRCH) [37,38]. Semantic weight factors are constructed to guide the optimization of the loss function and obtain better retrieval performance in Multiple Deep neural networks with Multiple labels for Cross-modal Hashing (MDMCH) [39].…”