Deep canonical correlation analysis with progressive and hypergraph learning for cross-modal retrieval

Shao, Jie; Wang, Leiquan; Zhao, Zhicheng; Su, Fei; Cai, Anni

doi:10.1016/j.neucom.2016.06.047

Cited by 32 publications

(9 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hardoon et al, overview the applications of Canonical Correlation Analysis (CCA), where CCA is introduced to project text and image features into a shared vector space by maximizing the cross relation [14]. Researchers then propose variants of CCA-based approaches [25,32,44]. Visual-semantic embedding (VSE) present by Frome et al, learns semantic relationships between labels and explicitly maps images into the semantic embedding space [11].…”

Section: Text and Image Matchingmentioning

confidence: 99%

FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval

Gao

Jin

Chen

et al. 2020

Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

In this paper, we address the text and image matching in crossmodal retrieval of the fashion industry. Different from the matching in the general domain, the fashion matching is required to pay much more aention to the fine-grained information in the fashion images and texts. Pioneer approaches detect the region of interests (i.e., RoIs) from images and use the RoI embeddings as image representations. In general, RoIs tend to represent the "object-level" information in the fashion images, while fashion texts are prone to describe more detailed information, e.g. styles, aributes. RoIs are thus not fine-grained enough for fashion text and image matching. To this end, we propose FashionBERT, which leverages patches as image features. With the pre-trained BERT model as the backbone network, FashionBERT learns high level representations of texts and images. Meanwhile, we propose an adaptive loss to trade off multitask learning in the FashionBERT modeling. Two tasks (i.e., text and image matching and cross-modal retrieval) are incorporated to evaluate FashionBERT. On the public dataset, experiments demonstrate FashionBERT achieves significant improvements in performances than the baseline and state-ofthe-art approaches. In practice, FashionBERT is applied in a concrete cross-modal retrieval application. We provide the detailed matching performance and inference efficiency analysis.

show abstract

Section: Text and Image Matchingmentioning

confidence: 99%

FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval

Gao

Jin

Chen

et al. 2020

Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Zhang et al [9] proposed a method named mixture of probabilistic CCA (MixPCCA) to model the nonlinear correlations between different modalities. Shao et al [28] presented hypergraph semantic embedding (HSE) approach to model latent semantics from text to regularize the deep CCA subspace. Wang et al [29] developed a novel correlation subspace learning method by integrating structured sparsity regularization and intra-modal information to achieve better performance.…”

Section: Cross-modal Retrievalmentioning

confidence: 99%

Adversarial Learning Based Semantic Correlation Representation for Cross-Modal Retrieval

Zhu¹,

Song²,

Wei³

et al. 2020

Preprint

View full text Add to dashboard Cite

With the rapid development of Internet and the widely usage of smart devices, massive multimedia data are generated, collected, stored and shared on the Internet. This trend makes cross-modal retrieval problem become a hot issue in this years. Many existing works pay attentions on correlation learning to generate a common subspace for cross-modal correlation measurement, and others uses adversarial learning technique to abate the heterogeneity of multi-modal data. However, very few works combine correlation learning and adversarial learning to bridge the inter-modal semantic gap and diminish cross-modal heterogeneity. This paper propose a novel cross-modal retrieval method, named ALSCOR, which is an end-to-end framework to integrate cross-modal representation learning, correlation learning and adversarial. CCA model, accompanied by two representation model, VisNet and TxtNet is proposed to capture non-linear correlation. Beside, intra-modal classifier and modality classifier are used to learn intra-modal discrimination and minimize the inter-modal heterogeneity. Comprehensive experiments are conducted on three benchmark datasets. The results demonstrate that the proposed ALSCOR has better performance than the state-of-the-arts.

show abstract

“…In this subspace, multi-modal data have the same dimension representation characteristics, hence they can directly measure the similarity of different modal data. For example, a series of algorithms [10], [11] based on canonical correlation analysis (CCA) [12] projected different modal data into a shared subspace via subspace mapping. Zhang and Chen [13] proposed kernal CCA (KCCA) which mapped data of different modalities into high-dimensional space to learn the semantic concepts corresponding to images and text.…”

Section: Introductionmentioning

confidence: 99%

Semantic Consistency Cross-Modal Retrieval With Semi-Supervised Graph Regularization

Zhang

2020

IEEE Access

View full text Add to dashboard Cite

Most of the existing cross-modal retrieval methods make use of labeled data to learn projection matrices for different modal data. These methods usually learn the original semantic space to bridge the heterogeneous gap, ignoring the rich semantic information contained in unlabeled data. Accordingly, a semantic consistency cross-modal retrieval with semi-supervised graph regularization (SCCMR) algorithm is proposed, which integrates the prediction of labels and the optimization of projection matrices into a unified framework to ensure that the solution obtained is globally optimal. At the same time, the method uses graph embedding to consider the nearest neighbors in the potential subspace of paired images and texts as well as images and texts with the same semantics. l 21-norm constraint is applied to the projection matrices to select the discriminative features for different modal data. The results show that our method outperforms several advanced methods on four commonly used cross-modal retrieval datasets.

show abstract

Deep canonical correlation analysis with progressive and hypergraph learning for cross-modal retrieval

Cited by 32 publications

References 25 publications

FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval

FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval

Adversarial Learning Based Semantic Correlation Representation for Cross-Modal Retrieval

Semantic Consistency Cross-Modal Retrieval With Semi-Supervised Graph Regularization

Contact Info

Product

Resources

About