Bridging Music and Image via Cross-Modal Ranking Analysis

Wu, Xixuan; Qiao, Yu; Wang, Xiaogang; Tang, Xiaoou

doi:10.1109/tmm.2016.2557722

Cited by 25 publications

(10 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Third, large music collections contain different modalities of information, i.e., audio, images, and text, and all these data are suitable to be exploited for genre classification. Several approaches dealing with different modalities have been proposed (Wu et al, 2016;Schedl et al, 2013). However, to the best of our knowledge, no multimodal approach based on deep learning architectures has been proposed for this Music Information Retrieval (MIR) task, neither for singlelabel nor multi-label classification.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Deep Learning for Music Genre Classification

Oramas¹,

Barbieri²,

Nieto³

et al. 2018

Transactions of the International Society for Music Information Retrieval

View full text Add to dashboard Cite

Music genre labels are useful to organize songs, albums, and artists into broader groups that share similar musical characteristics. In this work, an approach to learn and combine multimodal data representations for music genre classification is proposed. Intermediate representations of deep neural networks are learned from audio tracks, text reviews, and cover art images, and further combined for classification. Experiments on single and multi-label genre classification are then carried out, evaluating the effect of the different learned representations and their combinations. Results on both experiments show how the aggregation of learned representations from different modalities improves the accuracy of the classification, suggesting that different modalities embed complementary information. In addition, the learning of a multimodal feature space increases the performance of pure audio representations, which may be specially relevant when the other modalities are available for training, but not at prediction time. Moreover, a proposed approach for dimensionality reduction of target labels yields major improvements in multi-label classification not only in terms of accuracy, but also in terms of the diversity of the predicted genres, which implies a more fine-grained categorization. Finally, a qualitative analysis of the results sheds some light on the behavior of the different modalities on the classification task.

show abstract

Section: Introductionmentioning

confidence: 99%

Multimodal Deep Learning for Music Genre Classification

Oramas¹,

Barbieri²,

Nieto³

et al. 2018

Transactions of the International Society for Music Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Co-occurring changes in audio and video content of music videos can be detected, where the correlations can be used in cross-modal audio-visual music retrieval. Lyrics-based music attributes are utilized for image representation in [16]. Cross-modal ranking analysis is suggested to learn semantic similarity between music and image, with the aim of obtaining the optimal embedding spaces for music and image.…”

Section: B Cross-modal Music Retrievalmentioning

confidence: 99%

Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval

Tang

Raposo

et al. 2019

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

Deep cross-modal learning has successfully demonstrated excellent performances in cross-modal multimedia retrieval, with the aim of learning joint representations between different data modalities. Unfortunately, little research focuses on cross-modal correlation learning where temporal structures of different data modalities such as audio and lyrics are taken into account. Stemming from the characteristic of temporal structures of music in nature, we are motivated to learn the deep sequential correlation between audio and lyrics. In this work, we propose a deep cross-modal correlation learning architecture involving two-branch deep neural networks for audio modality and text modality (lyrics). Different modality data are converted to the same canonical space where inter modal canonical correlation analysis is utilized as an objective function to calculate the similarity of temporal structures. This is the first study on understanding the correlation between language and music audio through deep architectures for learning the paired temporal correlation of audio and lyrics. Pre-trained Doc2vec model followed by fully-connected layers (fully-connected deep neural network) is used to represent lyrics. Two significant contributions are made in the audio branch, as follows: i) pretrained CNN followed by fully-connected layers is investigated for representing music audio. ii) We further suggest an end-toend architecture that simultaneously trains convolutional layers and fully-connected layers to better learn temporal structures of music audio. Particularly, our end-to-end deep architecture contains two properties: simultaneously implementing feature learning and cross-modal correlation learning, and learning joint representation by considering temporal structures. Experimental results, using audio to retrieve lyrics or using lyrics to retrieve audio, verify the effectiveness of the proposed deep correlation learning architectures in cross-modal music retrieval.

show abstract

“…As online music streaming and video sharing websites arXiv:1801.02200v1 [cs.IR] 7 Jan 2018 have become increasingly popular, some research has been done on the relationship between music and album covers [1,2,3,4] and also on music and videos (instead of just images) as the visual modality [5,6,7,8] to explore the multimodal information present in both types of data. A recent study [9] also explored the cross-modal relations between the two modalities but using images with people talking and speech.…”

Section: Related Workmentioning

confidence: 99%

Cross-modal Embeddings for Video and Audio Retrieval

Surís

Duarte

Salvador

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The increasing amount of online videos brings several opportunities for training self-supervised neural networks. The creation of large scale datasets of videos such as the YouTube-8M allows us to deal with this large amount of data in manageable way. In this work, we find new ways of exploiting this dataset by taking advantage of the multi-modal information it provides. By means of a neural network, we are able to create links between audio and visual documents, by projecting them into a common region of the feature space, obtaining joint audio-visual embeddings. These links are used to retrieve audio samples that fit well to a given silent video, and also to retrieve images that match a given a query audio. The results in terms of Recall@K obtained over a subset of YouTube-8M videos show the potential of this unsupervised approach for cross-modal feature learning. We train embeddings for both scales and assess their quality in a retrieval problem, formulated as using the feature extracted from one modality to retrieve the most similar videos based on the features computed in the other modality.

show abstract

Bridging Music and Image via Cross-Modal Ranking Analysis

Cited by 25 publications

References 46 publications

Multimodal Deep Learning for Music Genre Classification

Multimodal Deep Learning for Music Genre Classification

Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval

Cross-modal Embeddings for Video and Audio Retrieval

Contact Info

Product

Resources

About