End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss

Dorfer, Matthias; Schlüter, Jan; Vall, Andreu; Korzeniowski, Filip; Widmer, Gerhard

doi:10.1007/s13735-018-0151-5

Cited by 43 publications

(29 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is exactly the dimension of our retrieval embedding space. At the top of the network we put a canonically correlated embedding layer (Dorfer et al, 2018) combined with the ranking loss described above. The structure of the model is analogous to the one presented in (Dorfer et al, 2017a) with the single difference that the sheet-image snippet is downsized by factor two (160 × 200 → 80 × 100) before being presented to the network.…”

Section: Embedding Space Learningmentioning

confidence: 99%

See 1 more Smart Citation

Learning Audio–Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification

Dorfer¹,

Hajič²,

Arzt³

et al. 2018

Transactions of the International Society for Music Information Retrieval

Self Cite

View full text Add to dashboard Cite

This work addresses the problem of matching musical audio directly to sheet music, without any higherlevel abstract representation. We propose a method that learns joint embedding spaces for short excerpts of audio and their respective counterparts in sheet music images, using multimodal convolutional neural networks. Given the learned representations, we show how to utilize them for two sheet-music-related tasks: (1) piece/score identification from audio queries and (2) retrieving relevant performances given a score as a search query. All retrieval models are trained and evaluated on a new, large scale multimodal audio-sheet music dataset which is made publicly available along with this article. The dataset comprises 479 precisely annotated solo piano pieces by 53 composers, for a total of 1,129 pages of music and about 15 hours of aligned audio, which was synthesized from these scores. Going beyond this synthetic training data, we carry out first retrieval experiments using scans of real sheet music of high complexity (e.g., nearly the complete solo piano works by Frederic Chopin) and commercial recordings by famous concert pianists. Our results suggest that the proposed method, in combination with the large-scale dataset, yields retrieval models that successfully generalize to data way beyond the synthetic training data used for model building.

show abstract

Section: Embedding Space Learningmentioning

confidence: 99%

“…This requires a different network architecture that can learn two separate projections, one for embedding the sheet music and one for embedding the audio, which can then be used independently of each other. For example, Dorfer, M., et al (2018). Learning Audio-Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification.…”

Section: Introductionmentioning

confidence: 99%

Learning Audio–Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification

Dorfer¹,

Hajič²,

Arzt³

et al. 2018

Transactions of the International Society for Music Information Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

“…It generalizes the model for uni-modal queries over textual corpora of webpages and documents, which has been the primary concern of IR systems for several decades [2]. Various systems that support non-textual retrieval have been proposed recently [2], [3]. Here, a user searches for information represented as images or audio based on queries that include several modalities beyond textual keywords.…”

Section: Introductionmentioning

confidence: 99%

“…Recently, first IR systems started to support the specification of multi-modal queries [3], [6]. Yet, they do not genuinely evaluate multi-modal queries, but consider them as a combination of several uni-modal queries, each of which covering a different modality.…”

Section: Introductionmentioning

confidence: 99%

Efficient and Effective Multi-Modal Queries Through Heterogeneous Network Embedding

Duong

Nguyen

Yin

et al. 2022

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

The heterogeneity of today's Web sources requires information retrieval (IR) systems to handle multi-modal queries. Such queries define a user's information needs by different data modalities, such as keywords, hashtags, user profiles, and other media.Recent IR systems answer such a multi-modal query by considering it as a set of separate uni-modal queries. However, depending on the chosen operationalisation, such an approach is inefficient or ineffective. It either requires multiple passes over the data or leads to inaccuracies since the relations between data modalities are neglected in the relevance assessment. To mitigate these challenges, we present an IR system that has been designed to answer genuine multi-modal queries. It relies on a heterogeneous network embedding, so that features from diverse modalities can be incorporated when representing both, a query and the data over which it shall be evaluated. By embedding a query and the data in the same vector space, the relations across modalities are made explicit and exploited for more accurate query evaluation. At the same time, multi-modal queries are answered with a single pass over the data. An experimental evaluation using diverse real-world and synthetic datasets illustrates that our approach returns twice the amount of relevant information compared to baseline techniques, while scaling to large multi-modal databases.Index Terms-query embedding, graph embedding, heterogeneous information network ✦ • A model for heterogeneous data: We introduce a representation of heterogeneous data based on HINs to capture the semantic relations between data of different modalities of the same entity [13], [14]. This includes a

show abstract

“…Choi et al [8] proposed a face identification method based on multi-class pairwise loss. Dorfer et al [9] proposed a multimodal retrieval method based on pairwise losses. Shi et al [10] proposed a deep ranking hash for histopathological retrieval and classification tasks.…”

Section: Introductionmentioning

confidence: 99%

An Adaptive and Asymmetric Residual Hash for Fast Image Retrieval

Cheng

Wang

2019

IEEE Access

View full text Add to dashboard Cite

Hashing algorithm has attracted great attention in recent years. In order to improve the query speed and retrieval accuracy, this paper proposes an adaptive and asymmetric residual hash (AASH) algorithm based on residual hash, integrated learning, and asymmetric pairwise loss. The specific description of the AASH algorithm is as follows: 1) the integrated learning model is proposed based on transfer learning and multi-feature fusion strategy to learn the database hash code; 2) the residual hash model is proposed based on ResNet-50 to learn the query image hash code; 3) the asymmetric pairwise loss is proposed and the parameters of the residual hash model is optimized based on the database hash code; 4) the algorithm learns the database hash code and the query image hash code in an asymmetric manner, and integrates the feature learning part and the hash-coded part in one frame. The experimental results on three different datasets fully demonstrate that the proposed AASH method has better performance than most symmetric and asymmetric deep hash algorithms. Specifically, the optimal result of the AASH algorithm is 0.971 on Cifar10 when the hyperparameter is 100 and the hash code length is 32. The optimal result of the AASH algorithm is 0.945 on ceil images when the hyperparameter is 10 and the hash code length is 24. The optimal result of the AASH algorithm is 0.945 on FD-XJ when the hyperparameter is 15 and the hash code length is 32. In addition, the algorithm verifies convergence, time loss, and effectiveness. INDEX TERMSResidual hash, asymmetric manner, adaptive and asymmetric residual hash (AASH), information search, hash coding.

show abstract

End-to-end cross-modality retrieval with CCA projections and pairwise ranking loss

Cited by 43 publications

References 16 publications

Learning Audio–Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification

Learning Audio–Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification

Efficient and Effective Multi-Modal Queries Through Heterogeneous Network Embedding

An Adaptive and Asymmetric Residual Hash for Fast Image Retrieval

Contact Info

Product

Resources

About