MTFH: A Matrix Tri-Factorization Hashing Framework for Efficient Cross-Modal Retrieval

Abstract

Hashing has recently sparked a great revolution in cross-modal retrieval due to its low storage cost and high query speed. Most existing cross-modal hashing methods learn unified hash codes in a common Hamming space to represent all multi-modal data and make them intuitively comparable. However, such unified hash codes could inherently sacrifice their representation scalability because the data from different modalities may not have one-to-one correspondence and could be stored more efficiently by different hash codes of unequal lengths. To mitigate this problem, this paper proposes a generalized and flexible cross-modal hashing framework, termed Matrix Tri-Factorization Hashing (MTFH), which not only preserves the semantic similarity between the multi-modal data points, but also works seamlessly in various settings including paired or unpaired multi-modal data, and equal or varying hash length encoding scenarios. Specifically, MTFH exploits an efficient objective function to jointly learn the flexible modality-specific hash codes with different length settings, while simultaneously excavating two semantic correlation matrices to ensure heterogeneous data comparable. As a result, the derived hash codes are more semantically meaningful for various challenging cross-modal retrieval tasks. Extensive experiments evaluated on public benchmark datasets highlight the superiority of MTFH under various retrieval scenarios and show its very competitive performance with the state-of-the-arts.

Deep Semantic-Preserving Reconstruction Hashing for Unsupervised Cross-Modal Retrieval

“…To test the effectiveness of the DSPRH algorithm, we repeated the DJSRH [ 12 ] and DSAH [ 13 ], because the DSPRH enlightenment idea originated from DJSRH [ 12 ] and DSAH [ 13 ]. Specifically, these baselines are described as follows: UKD [ 14 ] (CVPR 2020), SRCH [ 11 ] (IJCAI 2020), MGAH [ 33 ] (TMM 2020), UGACH [ 9 ] (AAAI 2018), DJSRH [ 12 ] (ICCV 2019), DSAH [ 13 ] (ICMR 2020), DBRC [ 28 ] (TMM 2019), CRB [ 22 ] (TIP 2019), DADH [ 15 ] (ICMR 2020), AGAH [ 16 ] (ICMR 2019), UDCMH [ 6 ] (IJCAI, 2018), SCH-GAN [ 34 ] (TOC, 2020); DSPOH [ 29 ] (TNNLS, 2019), SCRATCH [ 23 ] (TCSVT, 2019), MTFH [ 10 ] (TPAMI, 2019), EGDH [ 37 ] (IJCAI, 2019), DMFH [ 31 ] (TCSVT, 2020), BATCH [ 26 ] (TKDE, 2020), ATFH-N [ 25 ] (TETCI, 2020), MDCH [ 32 ] (TMM, 2020), CPAH [ 27 ] (TIP, 2020), MLCAH [ 30 ] (TMM, 2020) a d SSAH [ 5 ] (CVPR, 2018).…”

“…Zhang et al proposed unsupervised generative adversarial cross-modal hashing [ 9 ] to perform unsupervised representation learning. Liu et al proposed matrix tri-factorization hashing [ 10 ] to explore effective objective functions. Wang et al proposed semantic-rebased cross-modal hashing [ 11 ] to achieve unsupervised learning.…”

“…In the two modalities of image and text, image-text retrieval is mainly to realize the matching problem of image and text to explore the correspondence between image regions and sentence words. In order to further capture the modal semantic information, reduce the amount of calculation and accelerate the retrieval speed, the deep hash algorithm has become the mainstream method of image-text retrieval to measure modal semantic information in the Hamming space [ 9 , 10 , 11 ]. Figure 1 shows the overall architecture of cross-modal retrieval.…”

OD Matching of Metro IC Card Data Based on Analysis Function

“…Chen et al [2] analyzed the high-dimensional and multivariate metro data through the random matrix theory and then predicted the abnormal data of card swiping. rough the analysis of IC card data, Yu et al [3,4] classified the abnormal OD data of passengers into two categories: passenger anomaly and system anomaly.…”

Adversarial Learning Based Semantic Correlation Representation for Cross-Modal Retrieval

“…These multi-modal data are usually used to describe the same events, scenes or objects in our daily life, and users always have the need to search relative multimedia data by the queries of different modalities. This retrieval paradigm is called cross-modal retrieval [1][2][3], which attracts more and more attentions in the community of multimedia.…”