Differentiable Cross-modal Hashing via Multimodal Transformers

Tu, Junfeng; Liu, Xueliang; Lin, Zongxiang; Hong, Richang; Wang, Meng

doi:10.1145/3503161.3548187

Cited by 26 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since binary optimization is hard, most pair-wise methods employ a continuous relaxation strategy in training that will inevitably lead to quantization errors. To solve this problem, by introducing a selection mechanism, Tu et al proposed a novel Differentiable Cross-modal Hashing via Multimodal Transformers (DCHMT) algorithm to make binary optimization become differentiable [51]. Recently, Zhang et al introduced a Modality-Invariant Asymmetric Networks (MIAN) network to study asymmetric intra-modal and inter-modal similarity preservation methods in a probabilistic modality alignment framework [52].…”

Section: B Deep Supervised Cross-modal Hashingmentioning

confidence: 99%

“…Meanwhile, the Bag of Words (Bow) model is usually employed to denote the text [56], and then the MLP model is employed to extract the potential semantic information of the text. However, due to the sparsity nature of Bag-of-Words, previous studies have demonstrated that the above approach may ignore some valuable text information [11], [28], [51], leading to the generation of sub-optimal hash codes. Therefore, inspired by [32], the transformer encoders are introduced to achieve our proposed Image-Network and Text-Network, obtaining potential feature representations from image modalities and text modalities, respectively.…”

Section: Feature Learningmentioning

confidence: 99%

See 1 more Smart Citation

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Han,

Wang,

Chen

et al. 2024

IEEE Access

View full text Add to dashboard Cite

With the efficiency of storage and retrieval speed, the hashing methods have attracted a lot of attention for cross-modal retrieval applications. In contrast to traditional cross-modal hashing by using handcrafted features, deep cross-modal hashing integrates the advantages of deep learning and hashing methods to encode raw multimodal data into compact binary codes with semantic information preserved. Generally speaking, most of the existing deep cross-modal hashing methods simply define the semantic similarity between heterogeneous modalities by counting the number of shared semantic labels (such as, two samples share at least one label, they are similar, otherwise they are dissimilar), which fails to represent the accurate multi-label semantic relations between heterogeneous data. In this paper, we propose a new Deep Self-supervised Hashing with Fine-grained Similarity Mining (DSH-FSM) framework to efficiently preserve the fine-grained multi-label semantic similarity, learning a highly separable embedding space. Specifically, by employing an asymmetric guidance strategy, a novel Semantic-Network is introduced into cross-modal hashing to learn two semantic dictionaries, including the semantic feature dictionary and the semantic code dictionary, which guides the Image-Network and the Text-Network to capture multi-label semantic relevance across different modalities. Based on the obtained semantic dictionary, an asymmetric marginscalable loss is proposed to obtain fine-grained pair-wise similarity information, which could contribute to the production of similarity-preserving and discriminative binary codes. Besides, two feature extractors with transformer encoders are designed to achieve the Image-Network and Text-Network, which could extract the representative semantic characteristics from raw heterogeneous samples. Extensive experimental results on various benchmark datasets show that our proposed DSH-FSM framework achieves state-of-the-art crossmodal similarity search performance. Compared to the state-of-the-art methods, the results of mAP are significantly improved by 1.9%, 9.1%, and 9.8%, respectively, on the three widely used datasets.

show abstract

Section: B Deep Supervised Cross-modal Hashingmentioning

confidence: 99%

Section: Feature Learningmentioning

confidence: 99%

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Han,

Wang,

Chen

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…6) Transformer methods: Transformer methods [160], [161] leverage Transformer encoders to effectively capture fine-grained semantic information and introduce label information for supervision to model complex relationships within multi-modal data. For instance, Differentiable Cross-modal Hashing via Multi-modal Transformers (DCHMT) [160] constructs a multi-modal transformer to capture detailed crossmodal semantic information and introduces a micro-hashing module for mapping modal representations into hash codes.…”

Section: Supervised Cross-modal Hashing Retrievalmentioning

confidence: 99%

Efficient Query-based Black-box Attack against Cross-modal Hashing Retrieval

Zhu

Wang

et al. 2023

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Deep cross-modal hashing retrieval models inherit the vulnerability of deep neural networks. They are vulnerable to adversarial attacks, especially for the form of subtle perturbations to the inputs. Although many adversarial attack methods have been proposed to handle the robustness of hashing retrieval models, they still suffer from two problems: 1) Most of them are based on the white-box settings, which is usually unrealistic in practical application. 2) Iterative optimization for the generation of adversarial examples in them results in heavy computation. To address these problems, we propose an Efficient Query-based Black-Box Attack (EQB 2 A) against deep cross-modal hashing retrieval, which can efficiently generate adversarial examples for the black-box attack. Specifically, by sending a few query requests to the attacked retrieval system, the cross-modal retrieval model stealing is performed based on the neighbor relationship between the retrieved results and the query, thus obtaining the knockoffs to substitute the attacked system. A multi-modal knockoffs-driven adversarial generation is proposed to achieve efficient adversarial example generation. While the entire network training converges, EQB 2 A can efficiently generate adversarial examples by forward-propagation with only given benign images. Experiments show that EQB 2 A achieves superior attacking performance under the black-box setting.

show abstract

“…Compared with the shallow hash method of manually extracting features, the hash method based on deep learning greatly improves the distinctiveness and effectiveness of the extracted features.The Deep Cross-modal Hashing (DCMH) [8] first combined deep learning with hash learning, and proposed an end-to-end learning framework that integrates feature learning and hash learning. After this, many hash methods based on deep learning have been studied successively, such as [9,[14][15][16][17][18]. Most of the existing hash methods use the co-occurrence information of input image text pairs or artificially annotated semantic tags.…”

Section: Hash Learningmentioning

confidence: 99%

TECMH: Transformer-Based Cross-Modal Hashing For Fine-Grained Image-Text Retrieval

Ma¹,

Zheng²,

Ma³

et al. 2023

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

In recent years, cross-modal hash retrieval has become a popular research field because of its advantages of high efficiency and low storage. Cross-modal retrieval technology can be applied to search engines, crossmodal medical processing, etc. The existing main method is to use a multi-label matching paradigm to finish the retrieval tasks. However, such methods do not use fine-grained information in the multi-modal data, which may lead to suboptimal results. To avoid cross-modal matching turning into label matching, this paper proposes an end-to-end fine-grained cross-modal hash retrieval method, which can focus more on the fine-grained semantic information of multi-modal data. First, the method refines the image features and no longer uses multiple labels to represent text features but uses BERT for processing. Second, this method uses the inference capabilities of the transformer encoder to generate global fine-grained features. Finally, in order to better judge the effect of the fine-grained model, this paper uses the datasets in the image text matching field instead of the traditional label-matching datasets. This article experiment on Microsoft COCO (MS-COCO) and Flickr30K datasets and compare it with the previous classical methods. The experimental results show that this method can obtain more advanced results in the cross-modal hash retrieval field.

show abstract

Differentiable Cross-modal Hashing via Multimodal Transformers

Cited by 26 publications

References 25 publications

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Efficient Query-based Black-box Attack against Cross-modal Hashing Retrieval

TECMH: Transformer-Based Cross-Modal Hashing For Fine-Grained Image-Text Retrieval

Contact Info

Product

Resources

About