Multi-Task Music Representation Learning from Multi-Label Embeddings

Abstract

This paper presents a novel approach to music representation learning. Triplet loss based networks have become popular for representation learning in various multimedia retrieval domains. Yet, one of the most crucial parts of this approach is the appropriate selection of triplets, which is indispensable, considering that the number of possible triplets grows cubically. We present an approach to harness multitag annotations for triplet selection, by using Latent Semantic Indexing to project the tags onto a high-dimensional space. From this we estimate tag-relatedness to select hard triplets. The approach is evaluated in a multi-task scenario for which we introduce four large multi-tag annotations for the Million Song Dataset for the music properties genres, styles, moods, and themes.

Unsupervised cross-modal audio representation learning from unstructured multilingual text

“…Track-relatedness by Collection Membership: The first evaluation is based on common approaches to representation learning which use membership to a class [8], label [24] or identity [19,27] to select positive and negative examples for triplet based neural networks. As can be observed in Figure 5 this approach tends to learn representation which focus on collection related features and acoustic artifacts.…”

“…A differentiated evaluation of the learned semantic context as provided in this paper was missing. Concerning Representation Learning the presented paper in mostly related to [24] in which we extended the Million Song Dataset (MSD) [2] with additional ground truth multi-label assignments for Moods, Styles and Themes. Further, we extended the single-label Genre labels provided in [26] to multilabel assignments.…”

“…In [19] the authors defined music similarity on tracks originating from the same artists and used a triplet deep neural network architecture (see Section 1) to learn an optimized music optimization. Using identity or categorical data as ground-truth is sub-optimal [24] because it usually defines acoustic similarity too coarse. In [24] we learned a music representation from multi-label assignments using Latent Semantic indexing (LSI) to project the discrete information into a continuous space from which a track-similarity function based on tag-relatedness was derived.…”

“…Using identity or categorical data as ground-truth is sub-optimal [24] because it usually defines acoustic similarity too coarse. In [24] we learned a music representation from multi-label assignments using Latent Semantic indexing (LSI) to project the discrete information into a continuous space from which a track-similarity function based on tag-relatedness was derived. This tag-based track-similarity was then transferred to the audio space using a Triplet Network with a margin maximizing loss and online triplet selection strategy.…”

“…In this paper we build on the conclusions of [24] that by adding more content related information to the input space of the LSI projections, the definition of LSI topics improves, resulting in a better track-relatedness function. We adopt this approach and extend it from a discrete categorical space with a fixed vocabulary to an open, unstructured, multi-lingual free-text space.…”