Hierarchical semantic interaction-based deep hashing network for cross-modal retrieval

Abstract

Due to the high efficiency of hashing technology and the high abstraction of deep networks, deep hashing has achieved appealing effectiveness and efficiency for large-scale cross-modal retrieval. However, how to efficiently measure the similarity of fine-grained multi-labels for multi-modal data and thoroughly explore the intermediate layers specific information of networks are still two challenges for high-performance cross-modal hashing retrieval. Thus, in this paper, we propose a novel Hierarchical Semantic Interaction-based Deep Hashing Network (HSIDHN) for large-scale cross-modal retrieval. In the proposed HSIDHN, the multi-scale and fusion operations are first applied to each layer of the network. A Bidirectional Bi-linear Interaction (BBI) policy is then designed to achieve the hierarchical semantic interaction among different layers, such that the capability of hash representations can be enhanced. Moreover, a dual-similarity measurement (“hard” similarity and “soft” similarity) is designed to calculate the semantic similarity of different modality data, aiming to better preserve the semantic correlation of multi-labels. Extensive experiment results on two large-scale public datasets have shown that the performance of our HSIDHN is competitive to state-of-the-art deep cross-modal hashing methods.

Unsupervised Deep Cross-Modal Hashing Retrieval with Multiple Dense Networks

“…Compared with using one modality alone, combining multiple modalities, such as image and text, will help people gain more knowledge. Therefore, to search other relevant data of different modalities with query data, a high-performance and robust cross-modal information retrieval method is urgently needed [6,7]. Because the amount of data in different modalities is too large, one vital bottleneck restricting cross-modal retrieval is how to achieve effective retrieval and keep efficient storage.…”