Flexible Cross-Modal Hashing

Yu, Guoxian; Liu, Xuanwu; Wang, Jun; Domeniconi, Carlotta; Zhang, Xiangliang

doi:10.1109/tnnls.2020.3027729

Cited by 17 publications

(5 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To evaluate the performance of the proposed FMBSD method, we compare its experimental results with the results of several state-of-the-art methods of different types developed for realistic problems in practical scenario, containing WMCA (Lampert and Krömer, 2010), MMPDL (Liu et al, 2018), FlexCMH (Yu et al, 2020), PMH , GSPH (Mandal et al, 2019) and UCMH (Gao et al, 2020a), which are without pointwise correspondences (unpaired methods). The first two methods use latent subspace approaches across multimodal data, while the next methods are hashing-based ones.…”

Section: Comparison Methodsmentioning

confidence: 99%

“…These two types of correspondences, pointwise and batch correspondences, are very strong assumptions and cannot handle in many of realistic problems. In last recent years, a few works are exploited for the practical problems, such as UCMH (Gao et al, 2020a) which address the data with completely unpaired relationships by mapping data of different modalities to their respective semantic spaces, and FlexCMH (Yu et al, 2020) which introduces a clustering-based matching strategy to find the potential correspondences between samples across modalities.…”

Section: Multimodal Multiclass Classificationmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Behmanesh,

Adibi,

Chanussot

et al. 2021

Preprint

View full text Add to dashboard Cite

Multimodal manifold modeling methods extend the spectral geometry-aware data analysis to learning from several related and complementary modalities. Most of these methods work based on two major assumptions: 1) there are the same number of homogeneous data samples in each modality, and 2) at least partial correspondences between modalities are given in advance as prior knowledge. This work proposes two new multimodal modeling methods. The first method establishes a general analyzing framework to deal with the multimodal information problem for heterogeneous data without any specific prior knowledge. For this purpose, first, we identify the localities of each manifold by extracting local descriptors via spectral graph wavelet signatures (SGWS). Then, we propose a manifold regularization framework based on the functional mapping between SGWS descriptors (FMBSD) for finding the pointwise correspondences. The second method is a manifold regularized multimodal classification based on pointwise correspondences (M 2 CPC) used for the problem of multiclass classification of multimodal heterogeneous, which the correspondences between modalities are determined based on the FMBSD method. The experimental results of evaluating the FMBSD method on three common cross-modal retrieval datasets and evaluating the (M 2 CPC) method on three benchmark multimodal multiclass classification datasets indicate their effectiveness and superiority over state-of-the-art methods.

show abstract

Section: Comparison Methodsmentioning

confidence: 99%

Section: Multimodal Multiclass Classificationmentioning

confidence: 99%

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Behmanesh,

Adibi,

Chanussot

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, when we input key words to information retrieval systems, we expect the systems can efficiently find out related news/images/videos from database. Hence, many cross-modal hashing (CMH) methods have been proposed to deal with such tasks (Wang et al 2016;Jiang and Li 2017;Yu et al 2022a;Liu et al 2019a).…”

Section: Introductionmentioning

confidence: 99%

“…modal data. Unfortunately, most CMH methods solely mine the commonality (shared subspace) to learn hashing functions and assume balanced multi-modal data (Wang et al 2016;Jiang and Li 2017;Yu et al 2022a).…”

Section: Introductionmentioning

confidence: 99%

Long-Tail Cross Modal Hashing

Wang

Gao

Zhou

et al. 2023

AAAI

View full text Add to dashboard Cite

Existing Cross Modal Hashing (CMH) methods are mainly designed for balanced data, while imbalanced data with long-tail distribution is more general in real-world. Several long-tail hashing methods have been proposed but they can not adapt for multi-modal data, due to the complex interplay between labels and individuality and commonality information of multi-modal data. Furthermore, CMH methods mostly mine the commonality of multi-modal data to learn hash codes, which may override tail labels encoded by the individuality of respective modalities. In this paper, we propose LtCMH (Long-tail CMH) to handle imbalanced multi-modal data. LtCMH firstly adopts auto-encoders to mine the individuality and commonality of different modalities by minimizing the dependency between the individuality of respective modalities and by enhancing the commonality of these modalities. Then it dynamically combines the individuality and commonality with direct features extracted from respective modalities to create meta features that enrich the representation of tail labels, and binaries meta features to generate hash codes. LtCMH significantly outperforms state-of-the-art baselines on long-tail datasets and holds a better (or comparable) performance on datasets with balanced labels.

show abstract

“…This scenario is referred to as Unpaired Cross-Modal Retrieval (UCMR). Recently, some works have been proposed to address Semi-paired Cross-Modal Retrieval [26], [27], [10], [28], where only partial data correspondence is given. However, semi-paired hashing…”

mentioning

confidence: 99%

Adaptive Marginalized Semantic Hashing for Unpaired Cross-Modal Retrieval

Luo¹,

Zhang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

In recent years, Cross-Modal Hashing (CMH) has aroused much attention due to its fast query speed and efficient storage. Previous literatures have achieved promising results for Cross-Modal Retrieval (CMR) by discovering discriminative hash codes and modality-specific hash functions. Nonetheless, most existing CMR works are subjected to some restrictions: 1) It is assumed that data of different modalities are fully paired, which is impractical in real applications due to sample missing and false data alignment, and 2) binary regression targets including the label matrix and binary codes are too rigid to effectively learn semantic-preserving hash codes and hash functions. To address these problems, this paper proposes an Adaptive Marginalized Semantic Hashing (AMSH) method which not only enhances the discrimination of latent representations and hash codes by adaptive margins, but also can be used for both paired and unpaired CMR. As a two-step method, in the first step, AMSH generates semantic-aware modality-specific latent representations with adaptively marginalized labels, which enlarges the distances between different classes, and exploits the labels to preserve the inter-modal and intra-modal semantic similarities into latent representations and hash codes. In the second step, adaptive margin matrices are embedded into the hash codes, and enlarge the gaps between positive and negative bits, which improves the discrimination and robustness of hash functions. On this basis, AMSH generates similarity-preserving hash codes and robust hash functions without strict one-to-one data correspondence requirement. Experiments are conducted on several benchmark datasets to demonstrate the superiority and flexibility of AMSH over some state-of-the-art CMR methods.

show abstract

Flexible Cross-Modal Hashing

Cited by 17 publications

References 48 publications

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Long-Tail Cross Modal Hashing

Adaptive Marginalized Semantic Hashing for Unpaired Cross-Modal Retrieval

Contact Info

Product

Resources

About