Flexible Online Multi-modal Hashing for Large-scale Multimedia Retrieval

Lu, Xu; Zhu, Lei; Cheng, Zhiyong; Li, Jingjing; Nie, Xiushan; Zhang, Huaxiang

doi:10.1145/3343031.3350999

Cited by 110 publications

(31 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are also several multi-modal methods, which handle multi-view retrieval in an online scenario by fusing multiple modalities. For instance, Dynamic Multi-View Hashing (DMVH) [46] and Flexible Online Multi-modal Hashing (FOMH) [33]. Nevertheless, these multi-modal methods still cannot carry out the cross-modal search.…”

Section: Related Workmentioning

confidence: 99%

Label Embedding Online Hashing for Cross-Modal Retrieval

Wang

Luo

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Supervised cross-modal hashing has gained a lot of attention recently. However, most existing methods learn binary codes or hash functions in a batch-based scheme, which is inefficient in an online scenario, i.e., data points come in a streaming fashion. Online hashing is a promising solution; however, there still exist several challenges, e.g., how to effectively exploit semantic information, how to discretely solve the binary optimization problem, how to efficiently update hash codes and hash functions. To address these issues, in this paper, we propose a novel supervised online cross-modal hashing method, i.e., Label EMbedding ONline hashing, LEMON for short. It builds a label embedding framework including label similarity preserving and label reconstructing, which may generate discriminative binary codes and reduce the computational complexity. Furthermore, it not only preserves the pairwise similarity of incoming data, but also establishes a connection between newly coming data and existing data by the inner product minimization on a block similarity matrix. In the light of this, it can exploit more similarity information and make the optimization less sensitive to incoming data, leading to effective binary codes. In addition, we design a discrete optimization algorithm to solve the binary optimization problem without relaxation. Therefore, the quantization error can be reduced. Moreover, its computational complexity is only relevant to the size of incoming data, making it very efficient and scalable to large-scale datasets. Extensive experimental results on three benchmark datasets demonstrate that LEMON outperforms some state-of-the-art offline and online cross-modal hashing methods in terms of accuracy and efficiency. CCS CONCEPTS • Computing methodologies → Learning paradigms; • Information systems → Multimedia and multimodal retrieval.

show abstract

Section: Related Workmentioning

confidence: 99%

Label Embedding Online Hashing for Cross-Modal Retrieval

Wang

Luo

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…Despite their simplicity and flexibility, they need longer binary codes with reasonable performance, as compared to data-dependent ones. Many methods [6]- [8] tend to keep the neighborhood relations among the original samples mapped in a low-dimensional Hamming space. Thus, it is not strange that data-dependent counterparts become the main candidate for large-scale image retrieval [8]- [15].…”

Section: Introductionmentioning

confidence: 99%

Self-Collaborative Unsupervised Hashing for Large-Scale Image Retrieval

Zhao

Luo

2022

IEEE Access

View full text Add to dashboard Cite

Learning based hashing approaches have achieved considerable success in large-scale image retrieval due to the query effectiveness and efficiency. However, most studies highly rely on supervised knowledge like data labels, thus might fail in unsupervised setting. To address this issue, we propose a selfcollaborative unsupervised hashing method (SCUH), which jointly learns hashing function and virtual labels in a self-collaborative manner. Different from existing unsupervised counterparts, SCUH treats sample features and the learned virtual labels as two groups of different features in a fair view. This results in two groups of transformation matrices, which have proven beneficial for capturing diversity information of different features to better advance binary codes. Moreover, we devise an alternate optimization algorithm to solve SCUH, in which each sub-problem has a closed-form solution as a byproduct. Experiments of image retrieval on three datasets show that SCUH performs favorably over eight comparative methods including unsupervised counterparts.

show abstract

“…Cross-modal similarity retrieval has been a popular research topic [14,18,20,25,28,28,32,33,37,40] with the objective to search the semantic similar instances from different modalities. In a typical scenario, instances in one modality, e.g., images, are retrieved given a query from another modality, e.g., text.…”

Section: Introductionmentioning

confidence: 99%

“…In a typical scenario, instances in one modality, e.g., images, are retrieved given a query from another modality, e.g., text. Hashing based cross-modal retrieval methods [14,17,18,20,25,32,33,41] largely improve the retrieval efficiency on both speed and storage by mapping large-scale and high-dimensional multi-modal media data into a common Hamming space, where semantic similar instances from different modalities can be represented by similar hash codes and the correlations among these instances can be effectively measured with their Hamming distance. Since different modality instances are heterogeneous in terms of their feature representation and distribution, it is necessary for hashing based cross-modal retrieval methods to explore appropriate methodologies that the modality gap can be bridged.…”

Section: Introductionmentioning

confidence: 99%

Joint-modal Distribution-based Similarity Hashing for Large-scale Unsupervised Deep Cross-modal Retrieval

Song

Qian

Guan

et al. 2020

Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

135

View full text Add to dashboard Cite

Hashing-based cross-modal search which aims to map multiple modality features into binary codes has attracted increasingly attention due to its storage and search efficiency especially in largescale database retrieval. Recent unsupervised deep cross-modal hashing methods have shown promising results. However, existing approaches typically suffer from two limitations: (1) They usually learn cross-modal similarity information separately or in a redundant fusion manner, which may fail to capture semantic correlations among instances from different modalities sufficiently and effectively. (2) They seldom consider the sampling and weighting schemes for unsupervised cross-modal hashing, resulting in the lack of satisfactory discriminative ability in hash codes. To overcome these limitations, we propose a novel unsupervised deep cross-modal hashing method called Joint-modal Distributionbased Similarity Hashing (JDSH) for large-scale cross-modal retrieval. First, we propose a novel cross-modal joint-training method by constructing a joint-modal similarity matrix to fully preserve the cross-modal semantic correlations among instances. Second, we propose a sampling and weighting scheme termed the Distributionbased Similarity Decision and Weighting (DSDW) method for unsupervised cross-modal hashing, which is able to generate more discriminative hash codes by pushing semantic similar instance pairs closer and pulling semantic dissimilar instance pairs apart. The experimental results demonstrate the superiority of JDSH compared with several unsupervised cross-modal hashing methods on two public datasets NUS-WIDE and MIRFlickr.

show abstract

Flexible Online Multi-modal Hashing for Large-scale Multimedia Retrieval

Cited by 110 publications

References 29 publications

Label Embedding Online Hashing for Cross-Modal Retrieval

Label Embedding Online Hashing for Cross-Modal Retrieval

Self-Collaborative Unsupervised Hashing for Large-Scale Image Retrieval

Joint-modal Distribution-based Similarity Hashing for Large-scale Unsupervised Deep Cross-modal Retrieval

Contact Info

Product

Resources

About