Phoneme Level Lyrics Alignment and Text-Informed Singing Voice Separation

Abstract

The goal of singing voice separation is to recover the vocals signal from music mixtures. State-of-the-art performance is achieved by deep neural networks trained in a supervised fashion. Since training data are scarce and music signals are extremely diverse, it remains challenging to achieve high separation quality across various recording and mixing conditions as well as music styles. In this paper, we investigate to which extent the separation can be improved when lyrics transcripts are used as additional information. To this end, we propose a joint approach to phoneme level lyrics alignment and text-informed singing voice separation. It is based on DTW-attention, a new monotonic attention mechanism including a differentiable approximation of dynamic time warping. Experimental results show that the method can align phonemes with mixed singing voice with high precision given accurate transcripts. It also achieves competitive results on challenging word level alignment test sets using less training data than state-of-the-art methods. Sequential alignment and informed separation lead to improved separation quality according to objective measures. Text information helps preserving spectral phoneme properties in the separated voice signals.

Deep Learning Approaches in Topics of Singing Information Processing

“…Schulze-Forster et al [158] adapted the attention mechanism for allowing the use of weakly-labeled side information by learning lyrics alignment as a byproduct. Schulze-Forster et al [188] further considered aligned phoneme sequences from lyrics transcripts, in contrast to word sequences in [157], as an additional input to the separation model. They proposed a sequential framework where a phoneme alignment network is followed by a modified Open-Unmix architecture for singing voice separation such that it takes the aligned phoneme information as an additional input.…”

A survey on deep reinforcement learning for audio-based applications

“…GPT-4, a large language model, serves as the annotator, while the Whisper speech recognition model [84] assists in audio transcription. Leveraging the MTG-Jamendo dataset with 55,000 audio songs in various languages, the model requires no training and undergoes direct testing on multiple datasets, including Jamendo [172], Hansen [173], MUSDB [174], and DSing [175]. This combined approach not only transcribes lyrics in multiple languages but also contributes to reducing the Word Error Rate (WER) in English.…”

Investigating self-supervised learning for lyrics recognition

“…The MusDB dataset consists of 150 full-length audios with background music, including 86 music for the train set, 14 for the validation, and 50 for the test set. MusDB is only 10 hours long due to the limited commercial music, and the lyrics are manually aligned using [22].…”