Deep Unified Cross-Modality Hashing by Pairwise Data Alignment

Abstract

With the increasing amount of multimedia data, cross-modality hashing has made great progress as it achieves sub-linear search time and low memory space. However, due to the huge discrepancy between different modalities, most existing cross-modality hashing methods cannot learn unified hash codes and functions for modalities at the same time. The gap between separated hash codes and functions further leads to bad search performance. In this paper, to address the issues above, we propose a novel end-to-end Deep Unified Cross-Modality Hashing method named DUCMH, which is able to jointly learn unified hash codes and unified hash functions by alternate learning and data alignment. Specifically, to reduce the discrepancy between image and text modalities, DUCMH utilizes data alignment to learn an auxiliary image to text mapping under the supervision of image-text pairs. For text data, hash codes can be obtained by unified hash functions, while for image data, DUCMH first maps images to texts by the auxiliary mapping, and then uses the mapped texts to obtain hash codes. DUCMH utilizes alternate learning to update unified hash codes and functions. Extensive experiments on three representative image-text datasets demonstrate the superiority of our DUCMH over several state-of-the-art cross-modality hashing methods.

Multi-Manifold Deep Discriminative Cross-Modal Hashing for Medical Image Retrieval

“…To bridge modality gap, JDSH [8] exploits a joint-modal similarity matrix, while DUCMH [9] relies on data alignment and image-text data pairs. DGCPN [10] explores intrinsic semantic relationships with graph-neighbor coherence to avoid suboptimal retrieval Hamming space. With introducing knowledge distillation scheme, KDCMH [11] trains an unsupervised method as the teacher model used to provide distillation information to guide supervised method.…”

“…Hash lookup, a widely used retrieval protocol in hashingbased retrieval [10], considers the retrieved instance whose Hamming distance to the query is less than Hamming radius as a positive sample. When measuring the precision of hash lookup protocol, we hope that a good hashing method can retrieve as many positive samples as possible, i.e., when the query instance o i and the retrieved instance o j are true relevant, the probability of being judged as relevant should be as large as possible, denoted as:…”

Distributed Manifold Hashing for Image Set Classification and Retrieval

“…Hashing (Wang et al 2018a;Gao et al 2023;Kou et al 2022;Wang et al 2021) has gained increasing interests in many large-scale applications. Its primary goal is to encode • We propose a novel distributed manifold hashing (DMH) method for compact image set representation.…”

“…Hashing Hashing (Wang et al 2018a,a;Gao et al 2023;Kou et al 2022;Wang et al 2021) aims to learn compact hash code in Hamming space while preserving similarity, which can be particularly useful for efficient largescale search. Many existing hashing methods (Gong et al 2013;Shen et al 2015) employ machine learning technique to learn hash function and hash code.…”

Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding

“…As shown in Figure 1(a), only a short video segment semantically matches the query, while most of the video contents are queryirrelevant. Clearly, TSG tries to break through the barrier between computer vision and natural language processing techniques for more challenging cross-modal grounding (Li et al, ,a, 2022Wang and Shi, 2023;Wang et al, 2021aWang et al, , 2020c.…”