Joint-modal Distribution-based Similarity Hashing for Large-scale Unsupervised Deep Cross-modal Retrieval

Abstract

Hashing-based cross-modal search which aims to map multiple modality features into binary codes has attracted increasingly attention due to its storage and search efficiency especially in largescale database retrieval. Recent unsupervised deep cross-modal hashing methods have shown promising results. However, existing approaches typically suffer from two limitations: (1) They usually learn cross-modal similarity information separately or in a redundant fusion manner, which may fail to capture semantic correlations among instances from different modalities sufficiently and effectively. (2) They seldom consider the sampling and weighting schemes for unsupervised cross-modal hashing, resulting in the lack of satisfactory discriminative ability in hash codes. To overcome these limitations, we propose a novel unsupervised deep cross-modal hashing method called Joint-modal Distributionbased Similarity Hashing (JDSH) for large-scale cross-modal retrieval. First, we propose a novel cross-modal joint-training method by constructing a joint-modal similarity matrix to fully preserve the cross-modal semantic correlations among instances. Second, we propose a sampling and weighting scheme termed the Distributionbased Similarity Decision and Weighting (DSDW) method for unsupervised cross-modal hashing, which is able to generate more discriminative hash codes by pushing semantic similar instance pairs closer and pulling semantic dissimilar instance pairs apart. The experimental results demonstrate the superiority of JDSH compared with several unsupervised cross-modal hashing methods on two public datasets NUS-WIDE and MIRFlickr.

[Retracted] Deep Unsupervised Hashing for Large‐Scale Cross‐Modal Retrieval Using Knowledge Distillation Model

“…ere are a total of 8 unsupervised hashing methods, including four shallow cross-modal hashing methods and four deep cross-modal hashing methods. CVH [7], IMH [11], CMFH [24], and LSSH [26] are shallow methods, while DBRC [31], UDCMH [28], DJSRH [29], and JDSH [27] are deep methods. For fairness, the comparison method applies the same settings as in the original work.…”

“…ese methods cannot effectively capture the complex nonlinear mapping of different modal data to the hamming space, so many unsupervised cross-modal methods introduce deep neural networks into the learning of hash codes to construct a nonlinear mapping from data to hash codes. Computational Intelligence and Neuroscience [10,22,23,[27][28][29][30]. DBRC [31] proposes deep binary reconstruction cross-modal hashing to maintain consistency within and between modalities.…”

“…DJSRH [29] constructs a joint semantic similarity matrix based on the neighborhood information of different modalities and proposes deep joint semantic reconstruction hashing for cross-modal retrieval. JDSH [27] fully preserves the cross-modal semantic association between instances by constructing the joint-modal similarity matrix and similarity decision and weighted method based on distribution. UKD [30] uses output generated by unsupervised methods to guide supervisory methods and make use of teacher-student optimization for propagating knowledge.…”

Unsupervised Deep Cross-Modal Hashing by Knowledge Distillation for Large-scale Cross-modal Retrieval

“…DJSRH [34] proposes a cross-modal joint semantic similarity matrix, which maximally reconstructs the joint semantic relations in Hamming space. JDSH [17] fully preserves the cross-modal semantic association between instances by constructing the joint-modal similarity matrix and similarity decision and weighted method based on distribution. UKD [38] uses output generated by unsupervised methods to guide supervisory methods and make use of teacher-student optimization for propagating knowledge.…”

Deep Adversarial Quantization Network for Cross-Modal Retrieval

“…Cross-modal retrieval plays an important role in the abundant appearance of multimedia data on the Internet [1,2,3]. The cross-modal retrieval task aims to establish an information retrieval system, which can support querying across content domains, e.g., searching for the related texts through a query image.…”