Deep correlation for matching images and text

Yan, Fei; Mikolajczyk, Krystian

doi:10.1109/cvpr.2015.7298966

Cited by 370 publications

(234 citation statements)

References 15 publications

Supporting

Mentioning

233

Contrasting

Order By: Relevance

“…Assuming f and g are differentiable with respect to Θ f and Θ g (as is the case for neural networks), this allows to optimize the nonlinear transformations via gradient-based methods. Yan and Mikolajczyk [31] suggest the following procedure to utilize DCCA for cross-modality retrieval: first, 1 We understand the correlation of two vectors to be defined as corr(x, y) = i j corr(x i , y j ). [31].…”

Section: Deep Canonical Correlation Analysismentioning

confidence: 99%

“…Yan and Mikolajczyk [31] suggest the following procedure to utilize DCCA for cross-modality retrieval: first, 1 We understand the correlation of two vectors to be defined as corr(x, y) = i j corr(x i , y j ). [31]. Note that all processing steps below the solid line are performed after network optimization is complete neural networks f and g are trained using the TNO, with a and b representing different views of an entity (e.g.…”

Section: Deep Canonical Correlation Analysismentioning

confidence: 99%

“…The promising performance of their approach is also in line with the findings of Costa et al [23] who state the following two hypotheses regarding the properties of efficient crossmodal retrieval spaces: first, the embedding spaces should account for low-level cross-modal correlations and second, they should enable semantic abstraction. In [31], both properties are met by a deep neural network-learning abstract representations-that is optimized with DCCA ensuring highly correlated latent representations.…”

mentioning

confidence: 99%

“…We thus need to compute their partial derivatives with respect to the network's hidden representations x and y, i.e., allows incorporating domain knowledge into the loss function. On the other hand, DCCA is designed to maximize correlation-which has already proven to be useful for crossmodality retrieval [31]-but does not allow to use loss formulations specialized for the task at hand.…”

mentioning

confidence: 99%

“…In a different approach, Yan and Mikolajczyk [31] propose to learn a joint embedding of text and images using Deep canonical correlation analysis (DCCA) [2]. Instead of a pairwise ranking loss, DCCA directly optimizes the correlation of learned latent representations of the two views.…”

mentioning

confidence: 99%

See 4 more Smart Citations

End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss

Dorfer

Schlüter

Vall

et al. 2018

Int J Multimed Info Retr

View full text Add to dashboard Cite

Cross-modality retrieval encompasses retrieval tasks where the fetched items are of a different type than the search query, e.g., retrieving pictures relevant to a given text query. The state-of-the-art approach to cross-modality retrieval relies on learning a joint embedding space of the two modalities, where items from either modality are retrieved using nearest-neighbor search. In this work, we introduce a neural network layer based on canonical correlation analysis (CCA) that learns better embedding spaces by analytically computing projections that maximize correlation. In contrast to previous approaches, the CCA layer allows us to combine existing objectives for embedding space learning, such as pairwise ranking losses, with the optimal projections of CCA. We show the effectiveness of our approach for cross-modality retrieval on three different scenarios (text-to-image, audio-sheet-music and zero-shot retrieval), surpassing both Deep CCA and a multi-view network using freely learned projections optimized by a pairwise ranking loss, especially when little training data is available (the code for all three methods is released at: https://github.com/CPJKU/cca_layer).

show abstract

Section: Deep Canonical Correlation Analysismentioning

confidence: 99%

Section: Deep Canonical Correlation Analysismentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss

Dorfer

Schlüter

Vall

et al. 2018

Int J Multimed Info Retr

View full text Add to dashboard Cite

show abstract

Finding Homologous Points

2017

Theory and Applications of Image Registration

View full text Add to dashboard Cite

Cross‐modal retrieval with dual multi‐angle self‐attention

Zheng

Zhang

et al. 2020

Asso for Info Science & Tech

View full text Add to dashboard Cite

In recent years, cross‐modal retrieval has been a popular research topic in both fields of computer vision and natural language processing. There is a huge semantic gap between different modalities on account of heterogeneous properties. How to establish the correlation among different modality data faces enormous challenges. In this work, we propose a novel end‐to‐end framework named Dual Multi‐Angle Self‐Attention (DMASA) for cross‐modal retrieval. Multiple self‐attention mechanisms are applied to extract fine‐grained features for both images and texts from different angles. We then integrate coarse‐grained and fine‐grained features into a multimodal embedding space, in which the similarity degrees between images and texts can be directly compared. Moreover, we propose a special multistage training strategy, in which the preceding stage can provide a good initial value for the succeeding stage and make our framework work better. Very promising experimental results over the state‐of‐the‐art methods can be achieved on three benchmark datasets of Flickr8k, Flickr30k, and MSCOCO.

show abstract

Deep correlation for matching images and text

Cited by 370 publications

References 15 publications

End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss

End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss

Finding Homologous Points

Cross‐modal retrieval with dual multi‐angle self‐attention

Contact Info

Product

Resources

About