Contrastive Learning with Positive-Negative Frame Mask for Music Representation

Abstract

Self-supervised learning, especially contrastive learning, has made an outstanding contribution to the development of many deep learning research fields. Recently, researchers in the acoustic signal processing field noticed its success and leveraged contrastive learning for better music representation. Typically, existing approaches maximize the similarity between two distorted audio segments sampled from the same music. In other words, they ensure a semantic agreement at the music level. However, those coarse-grained methods neglect some inessential or noisy elements at the frame level, which may be detrimental to the model to learn the effective representation of music. Towards this end, this paper proposes a novel Positive-nEgative frame mask for Music Representation based on the contrastive learning framework, abbreviated as PEMR. Concretely, PEMR incorporates a Positive-Negative Mask Generation module, which leverages transformer blocks to generate frame masks on Log-Mel spectrogram. We can generate self-augmented negative and positive samples by masking important components or inessential components, respectively. We devise a novel contrastive learning objective to accommodate both self-augmented positives/negatives sampled from the same music. We conduct experiments on four public datasets. The experimental results of two music-related downstream tasks, music classification and cover song identification, demonstrate the generalization ability and transferability of music representation learned by PEMR. CCS CONCEPTS• Computing methodologies → Unsupervised learning; Learning latent representations.

Contrastive Balancing Representation Learning for Heterogeneous Dose-Response Curves Estimation

“…Contrastive Representation Learning. Contrastive representation learning (Chen et al 2020;He et al 2020;Grill et al 2020;Zhang et al 2021Zhang et al , 2022Yao et al 2022;Gan et al 2023) is a self-supervised learning method. It approximates the latent representations by constructing contrastive samples (positive and negative instances) to facilitate instance discrimination (Wu et al 2018).…”

Multimodal Pretraining and Generation for Recommendation: A Tutorial

“…3) Audio-based pretraining, which has been studied in the context of music recommendation and retrieval. They are used to extract latent music representations to enhance recommendation and retrieval tasks, including Mu-sicBert [61], MART [52], PEMR [51], and UAE [6]. 4) Multimodal pretraining that aims to achieve multimodal content understanding and cross-modal alignment.…”

DisCover: Disentangled Music Representation Learning for Cover Song Identification

“…Moreover, users humming the query songs might not necessarily know/provide the meta-information. Overall, CSI as a challenging task has long attracted lots of researchers due to its potential applications in music representation learning [25,57], retrieval [32,46,61] and recommendation [14,35]. However, those cover songs may differ from the original song in key transposition, speed change, and structural variations, which challenges identifying the cover song.…”

“…As the labeled datasets on which supervised learning methods require extensive manual labeling, it is often costly and time-consuming, leading to limitations in the performance of supervised learning methods. For this reason, some audio researchers have adopted a self-supervised learning approach to learning musical representations [41,47,57,67]. For example, MusicBERT [67] models music self-representation with a multi-task learning framework.…”

“…For example, MusicBERT [67] models music self-representation with a multi-task learning framework. PEMR [57] proposes a positive-negative frame mask for music representation with contrastive learning. Many approaches to music representation learning focus on key pieces of music, while CSI focuses more on the whole song.…”