Self-Supervised Contrastive Learning for Singing Voices

Abstract

This study introduces self-supervised contrastive learning to acquire feature representations of singing voices. To acquire robust representations in an unsupervised manner, regular self-supervised contrastive learning trains neural networks to make the feature representation of a sample close to those of its computationally transformed versions. Similarly, we employ two transformations-pitch shifting and time stretching-considering the nature of singing voices. Nevertheless, we use them reversely: we train networks to push away representations of the transformed versions. The networks then attempt to discriminate changes in vocal timbres introduced by pitch shifting without time stretching and those in singing expressions introduced by time stretching without pitch shifting. Consequently, the acquired representations become attentive to vocal timbre and singing expression. This was confirmed through a singer identification task, where we trained a classifier to learn the relationship between the feature representations to the corresponding singer labels of 500 singers. As a result, the employed transformations helped the classifier improve the classification accuracy by 9.12% (top-1 accuracy: 63.08%) compared with the case where the feature representations fed to the classifier were acquired without the transformations (top-1 accuracy: 53.96%). Furthermore, the proposed approach can be extended to acquire feature representations attentive to either vocal timbre or singing expression but not to the other by changing how the transformations are incorporated. We particularly explored the characteristics of such vocal timbre-or singing expression-oriented feature representations against song genre, singer gender, and vocal technique, and confirmed that they successfully capture different aspects of singing voices.

Multi-Source Contrastive Learning from Musical Audio

“…The proposed framework also utilizes semantic information, by taking into account the existence or absence of particular sources in each excerpt. This diverges from a number of research works in the literature that utilize a specific source, such as the singing voice [17][18][19] or percussion [20,21]. Application of our proposed methodology on top of COLA [7], a popular selfsupervised audio representation learning framework indi- cates that it can yield competitive results to a number of contrastive pair creation strategies [9,10] in three different downstream tasks.…”

“…In [28], additive mixtures of audio excerpts are used in conjuction with their components as input pairs, while in [22], the authors expand on their previous work [28] by using an unsupervised source separation system [29] to extract separated views of the initial audio segment. In music signal processing, similarities between song excerpts and the corresponding vocals have been exploited, using triplet losses [17,18] or batch-wise contrastive losses [19], usually for the task of artist identification. Furthermore, in [20] the anchor-positive pairs are created using the percussive part of a music segment and its non-percussive accompaniment, without temporal cropping -an idea that was expanded in [21].…”

Identification of Non-Speaking and Minimal-Speaking Individuals Using Nonverbal Vocalizations

“…Nevertheless, the NMS community is typically overlooked in research focusing on the prevalent voice-based approach for person identification (PID). The predominant focus in voice-based PID studies lies in utilizing verbal vocalizations, specifically speech for speaker identification [3]- [6], and singing voice [7]- [12] for singer identification. In contrast, limited exploration exists regarding nonverbal vocalizations, albeit from speaking individuals, for PID.…”