Semi-Supervised Knowledge Distillation for Cross-Modal Hashing

Su, Mingyue; Gu, Guanghua; Ren, Xianlong; Fu, Hao; Zhao, Yao

doi:10.1109/tmm.2021.3129623

Cited by 14 publications

(5 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, by combining an intra-modal graph and a cross-modal graph, Wu et al proposed a Modalityspecific and Cross-modal Graph Convolutional Networks (MCGCN) approach to fully explore modality-specific semantic information and modality-shared semantic information [17]. Subsequently, Su et al proposed a semi-supervised knowledge distillation for cross-modal hashing (SKDCH) algorithm, in which teacher-student optimization is employed to propagate semantic knowledge resulting in satisfactory retrieval performance [19].…”

Section: Deep Semi-supervised Cross-modal Hashingmentioning

confidence: 99%

See 1 more Smart Citation

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Han,

Wang,

Chen

et al. 2024

IEEE Access

View full text Add to dashboard Cite

With the efficiency of storage and retrieval speed, the hashing methods have attracted a lot of attention for cross-modal retrieval applications. In contrast to traditional cross-modal hashing by using handcrafted features, deep cross-modal hashing integrates the advantages of deep learning and hashing methods to encode raw multimodal data into compact binary codes with semantic information preserved. Generally speaking, most of the existing deep cross-modal hashing methods simply define the semantic similarity between heterogeneous modalities by counting the number of shared semantic labels (such as, two samples share at least one label, they are similar, otherwise they are dissimilar), which fails to represent the accurate multi-label semantic relations between heterogeneous data. In this paper, we propose a new Deep Self-supervised Hashing with Fine-grained Similarity Mining (DSH-FSM) framework to efficiently preserve the fine-grained multi-label semantic similarity, learning a highly separable embedding space. Specifically, by employing an asymmetric guidance strategy, a novel Semantic-Network is introduced into cross-modal hashing to learn two semantic dictionaries, including the semantic feature dictionary and the semantic code dictionary, which guides the Image-Network and the Text-Network to capture multi-label semantic relevance across different modalities. Based on the obtained semantic dictionary, an asymmetric marginscalable loss is proposed to obtain fine-grained pair-wise similarity information, which could contribute to the production of similarity-preserving and discriminative binary codes. Besides, two feature extractors with transformer encoders are designed to achieve the Image-Network and Text-Network, which could extract the representative semantic characteristics from raw heterogeneous samples. Extensive experimental results on various benchmark datasets show that our proposed DSH-FSM framework achieves state-of-the-art crossmodal similarity search performance. Compared to the state-of-the-art methods, the results of mAP are significantly improved by 1.9%, 9.1%, and 9.8%, respectively, on the three widely used datasets.

show abstract

Section: Deep Semi-supervised Cross-modal Hashingmentioning

confidence: 99%

“…To reduce the cost of data annotation, the semi-supervised cross-modal hashing method is proposed by utilizing both labeled data and unlabeled data [17]- [19]. Moreover, to avoid incomplete and insufficient labels of the training data, some researchers propose the weakly-supervised cross-modal hashing method [20], [21].…”

Section: Introductionmentioning

confidence: 99%

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Han,

Wang,

Chen

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…constraints. Model compression is an effective method to optimize models, mainly divided into knowledge distillation ( [1], [2], [3], [4], [5]), pruning ( [6]) and quantization ( [7], [8]), etc. Through model compression, large models are effectively transformed into lightweight counterparts, facilitating their migration to mobile devices.…”

Section: Introductionmentioning

confidence: 99%

Category Contrastive Distillation with Self-Supervised Classification

Chen,

Xu,

Zheng

et al. 2024

Preprint

View full text Add to dashboard Cite

Knowledge distillation aims to transfer the knowledge from one model to another, which is widely used in model compression and transfer learning. Recent knowledge distillation algorithms mainly focused on the inter-sample or intra-sample knowledge, which haven't efficiently exploited supervised labels. In this work, we introduce the self-supervised tasks into conventional knowledge distillation framework to construct a new category-based knowledge distillation algorithm. To be specific, two new memory banks are constructed to store the category embedding predicted by the teacher and student models, respectively. Then, the student model can absorb the knowledge from the teacher's memory bank instead of the model itself. It utilizes the cross-batch knowledge based on the contrast of category embedding among various batches.Moreover, additional self-supervised tasks with various data augmentation strategies are are designed to combine with the knowledge distillation process, which guides the model training in a multi-task manner. In the experiments, our proposed algorithm outperforms most of the sample-based knowledge distillation on both CIFAR-100 and ImageNet datasets.

show abstract

“…In such methods, the aim is to learn a function that can transform different modalities into a common feature space [4,5], where we can compare them directly. Due to the quick expansion of the data scale and the decline of data retrieval efficiency, the hashing codes are applied to cross-modal retrieval tasks [6][7][8]. This type of method maps high-dimensional features to the Hamming space by transforming data into hash binary codes and uses XOR of hash binary codes to calculate the Hamming distance.…”

Section: Introductionmentioning

confidence: 99%

Cross-modal retrieval based on multi-dimensional feature fusion hashing

Ren,

2024

Front. Phys.

View full text Add to dashboard Cite

Along with the continuous breakthrough and popularization of information network technology, multi-modal data, including texts, images, videos, and audio, is growing rapidly. We can retrieve different modal data to meet our needs, so cross-modal retrieval has important theoretical significance and application value. In addition, because the data of different modalities can be mutually retrieved by mapping them to a unified Hamming space, hash codes have been extensively used in the cross-modal retrieval field. However, existing cross-modal hashing models generate hash codes based on single-dimension data features, ignoring the semantic correlation between data features in different dimensions. Therefore, an innovative cross-modal retrieval method using Multi-Dimensional Feature Fusion Hashing (MDFFH) is proposed. To better get the image’s multi-dimensional semantic features, a convolutional neural network, and Vision Transformer are combined to construct an image multi-dimensional fusion module. Similarly, we apply the multi-dimensional text fusion module to the text modality to obtain the text’s multi-dimensional semantic features. These two modules can effectively integrate the semantic features of data in different dimensions through feature fusion, making the generated hash code more representative and semantic. Extensive experiments and corresponding analysis results on two datasets indicate that MDFFH’s performance outdoes other baseline models.

show abstract

Semi-Supervised Knowledge Distillation for Cross-Modal Hashing

Cited by 14 publications

References 36 publications

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Category Contrastive Distillation with Self-Supervised Classification

Cross-modal retrieval based on multi-dimensional feature fusion hashing

Contact Info

Product

Resources

About