Deep Multi-Level Semantic Hashing for Cross-Modal Retrieval

Abstract

With the rapid growth of multimodal data, the cross-modal search has widely attracted research interests. Due to its efficiency on storage and computing, hashing-based methods are broadly used for large scale cross-modal retrieval. Most existing hashing methods are designed based on binary supervision, which transforms complex relationships of multi-label data into simple similar or dissimilar. However, few methods have explored the rich semantic information implicit in multi-label data to improve the accuracy of searching results. In this paper, the multi-level semantic supervision generating approach is proposed by exploring the label relevance. And a deep hashing framework is designed for multi-label image-text cross retrieval tasks. It can simultaneously capture the binary similarity and the complex multi-level semantic structure of data in different forms. Moreover, the effects of three different convolutional neural networks, CNN-F, VGG-16, and ResNet-50, on the retrieval results are compared. The experimental results on an open source cross-modal dataset show that our approach outperforms several state-of-the-art hashing methods, and the retrieval result on the CNN-F network is better than VGG-16 and ResNet-50. INDEX TERMS Cross-modal retrieval, deep learning, hashing method, multi-label learning.

Comparative analysis on cross-modal information retrieval: A review

“…So, it may perform poorly in the case of less labeled data. In [129], authors have proposed a multi-level semantic supervision generating method after exploring the label relevance, and a deep hashing framework is introduced for multi-label imagetext cross-modal retrieval. It can capture the binary similarity as well as the complex multi-label semantic structure of data in diverse forms at the same time.…”

BRS cS: a hybrid recommendation model fusing multi-source heterogeneous data

“…Currently, the item-based collaborative filtering algorithm is used broadly in the industry, which recommends items similar to the items liked by users to them [16]. The model-based collaborative filtering algorithms mainly recommend items to users through machine learning and data mining models [17].…”

Greedy Learning of Deep Boltzmann Machine (GDBM)’s Variance and Search Algorithm for Efficient Image Retrieval

“…Reference [56] constructed an approach based on multilevel semantic supervision generation by discovering label relevance. This paper had been designed with a deep hashing structure in order to cross retrieve the multi-label images with text.…”