Weakly-supervised Cross-modal Hashing

Liu, Xuanwu; Yu, Guoxian; Domeniconi, Carlotta; Wang, Jun; Xiao, Guoqiang; Guo, Maozu

doi:10.1109/tbdata.2019.2954516

Cited by 15 publications

(3 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Without loss of generality, we firstly present LtCMH based on two modalities (image and text). LtCMH can also be applied to other data modality or extended to ≥ 3 modalities (Liu et al 2022)…”

Section: The Proposed Methodology Problem Overviewmentioning

confidence: 99%

Long-Tail Cross Modal Hashing

Wang

Gao

Zhou

et al. 2023

AAAI

View full text Add to dashboard Cite

Existing Cross Modal Hashing (CMH) methods are mainly designed for balanced data, while imbalanced data with long-tail distribution is more general in real-world. Several long-tail hashing methods have been proposed but they can not adapt for multi-modal data, due to the complex interplay between labels and individuality and commonality information of multi-modal data. Furthermore, CMH methods mostly mine the commonality of multi-modal data to learn hash codes, which may override tail labels encoded by the individuality of respective modalities. In this paper, we propose LtCMH (Long-tail CMH) to handle imbalanced multi-modal data. LtCMH firstly adopts auto-encoders to mine the individuality and commonality of different modalities by minimizing the dependency between the individuality of respective modalities and by enhancing the commonality of these modalities. Then it dynamically combines the individuality and commonality with direct features extracted from respective modalities to create meta features that enrich the representation of tail labels, and binaries meta features to generate hash codes. LtCMH significantly outperforms state-of-the-art baselines on long-tail datasets and holds a better (or comparable) performance on datasets with balanced labels.

show abstract

Section: The Proposed Methodology Problem Overviewmentioning

confidence: 99%

Long-Tail Cross Modal Hashing

Wang

Gao

Zhou

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

“…Considering the issue of insufficient labels and incomplete labels, the weakly-supervised method is proposed. For instance, Liu et al proposed a weakly supervised cross-modal hashing (WCHash) framework in which a multi-label weak labels scheme is adopted to enrich the labels of the training data [20].…”

Section: Deep Weakly-supervised Cross-modal Hashingmentioning

confidence: 99%

“…To reduce the cost of data annotation, the semi-supervised cross-modal hashing method is proposed by utilizing both labeled data and unlabeled data [17]- [19]. Moreover, to avoid incomplete and insufficient labels of the training data, some researchers propose the weakly-supervised cross-modal hashing method [20], [21]. Meanwhile, by employing the predefined similarity measures or mining the intrinsic distributions, unsupervised cross-modal hashing is proposed to build hash projection functions and perform compact representation learning [22]- [27].…”

Section: Introductionmentioning

confidence: 99%

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Han,

Wang,

Chen

et al. 2024

IEEE Access

View full text Add to dashboard Cite

With the efficiency of storage and retrieval speed, the hashing methods have attracted a lot of attention for cross-modal retrieval applications. In contrast to traditional cross-modal hashing by using handcrafted features, deep cross-modal hashing integrates the advantages of deep learning and hashing methods to encode raw multimodal data into compact binary codes with semantic information preserved. Generally speaking, most of the existing deep cross-modal hashing methods simply define the semantic similarity between heterogeneous modalities by counting the number of shared semantic labels (such as, two samples share at least one label, they are similar, otherwise they are dissimilar), which fails to represent the accurate multi-label semantic relations between heterogeneous data. In this paper, we propose a new Deep Self-supervised Hashing with Fine-grained Similarity Mining (DSH-FSM) framework to efficiently preserve the fine-grained multi-label semantic similarity, learning a highly separable embedding space. Specifically, by employing an asymmetric guidance strategy, a novel Semantic-Network is introduced into cross-modal hashing to learn two semantic dictionaries, including the semantic feature dictionary and the semantic code dictionary, which guides the Image-Network and the Text-Network to capture multi-label semantic relevance across different modalities. Based on the obtained semantic dictionary, an asymmetric marginscalable loss is proposed to obtain fine-grained pair-wise similarity information, which could contribute to the production of similarity-preserving and discriminative binary codes. Besides, two feature extractors with transformer encoders are designed to achieve the Image-Network and Text-Network, which could extract the representative semantic characteristics from raw heterogeneous samples. Extensive experimental results on various benchmark datasets show that our proposed DSH-FSM framework achieves state-of-the-art crossmodal similarity search performance. Compared to the state-of-the-art methods, the results of mAP are significantly improved by 1.9%, 9.1%, and 9.8%, respectively, on the three widely used datasets.

show abstract

Dynamical User Intention Prediction via Multi-modal Learning

Liu

Mao

et al. 2020

Database Systems for Advanced Applications

View full text Add to dashboard Cite

Weakly-supervised Cross-modal Hashing

Cited by 15 publications

References 37 publications

Long-Tail Cross Modal Hashing

Long-Tail Cross Modal Hashing

Deep Self-Supervised Hashing With Fine-Grained Similarity Mining for Cross-Modal Retrieval

Dynamical User Intention Prediction via Multi-modal Learning

Contact Info

Product

Resources

About