Joint Singing Voice Separation and F0 Estimation with Deep U-Net Architectures

Abstract

Citation: Jansson, A., Bittner, R. M., Ewert, S. and Weyde, T. ORCID: 0000-0001- 8028-9905 (2019). Joint singing voice separation and F0 estimation with deep U-net architectures.Abstract-Vocal source separation and fundamental frequency estimation in music are tightly related tasks. The outputs of vocal source separation systems have previously been used as inputs to vocal fundamental frequency estimation systems; conversely, vocal fundamental frequency has been used as side information to improve vocal source separation. In this paper, we propose several different approaches for jointly separating vocals and estimating fundamental frequency. We show that joint learning is advantageous for these tasks, and that a stacked architecture which first performs vocal separation outperforms the other configurations considered. Furthermore, the best joint model achieves state-of-the-art results for vocal-f0 estimation on the iKala dataset. Finally, we highlight the importance of performing polyphonic, rather than monophonic vocal-f0 estimation for many real-world cases.

Content Based Singing Voice Extraction from a Musical Mixture

“…We use this prediction along with the vocoder features to synthesise the audio signal. We tried both the discrete representation of the fundamental frequency as described in [23] and a continuous representation, normalised to the range 0 to 1 as used in [13] and found that while the discrete representation leads to slightly higher accuracy in the output, the continuous representation produces a pitch contour perceptually more suitable for synthesis of the signal. Fig.…”

Vocal Melody Extraction via HRNet-Based Singing Voice Separation and Encoder-Decoder-Based F0 Estimation

“…Most recently, Nakano et al [22] and Jansson et al [23] almost at the same time proposed to train the SVS task and the VME task jointly. Both methods obtained promising results.…”

“…Both methods obtained promising results. In [22], a joint U-Net model stacking SVS and VME was proposed. However, limited by the size of datasets containing both pure vocal tracks and their corresponding F0 annotations, the authors used a large internal dataset where reference F0 values were annotated by the VME method Deep Salience [5].…”

Environmental sound processing and its applications

“…Therefore, it is essential to accurately estimate the Wiener gain in each time‐frequency slot using a method such as time‐frequency masking. Approaches to directly estimate Wiener gain have also been developed, which use deep learning techniques to model a mapping function from the mixed sound signals into time‐frequency masks, and deep networks pretrained with target sound‐source signals . On the other hand, deep clustering has been proposed as an approach for estimating not the time‐frequency mask, but time‐frequency embedding vectors, so that the embedding vectors for time‐frequency slot pairs dominated by the same sound‐source signal are close together, while those for others signals are further away.…”