Monaural Singing Voice and Accompaniment Separation Based on Gated Nested U-Net Architecture

Abstract

This paper proposes a separation model adopting gated nested U-Net (GNU-Net) architecture, which is essentially a deeply supervised symmetric encoder–decoder network that can generate full-resolution feature maps. Through a series of nested skip pathways, it can reduce the semantic gap between the feature maps of encoder and decoder subnetworks. In the GNU-Net architecture, only the backbone not including nested part is applied with gated linear units (GLUs) instead of conventional convolutional networks. The outputs of GNU-Net are further fed into a time-frequency (T-F) mask layer to generate two masks of singing voice and accompaniment. Then, those two estimated masks along with the magnitude and phase spectra of mixture can be transformed into time-domain signals. We explored two types of T-F mask layer, discriminative training network and difference mask layer. The experiment results show the latter to be better. We evaluated our proposed model by comparing with three models, and also with ideal T-F masks. The results demonstrate that our proposed model outperforms compared models, and it’s performance comes near to ideal ratio mask (IRM). More importantly, our proposed model can output separated singing voice and accompaniment simultaneously, while the three compared models can only separate one source with trained model.

3 directional Inception-ResUNet: Deep spatial feature learning for multichannel singing voice separation with distortion

“…Jansson et al [ 4 ] components an audio signal by converting it to an image, processing it with a U-Net neural network, and storing the resulting spectral mask. GNU-Net [ 10 ] leverages a supervised symmetric encoder-decoder architecture for generating full-resolution feature maps. SVSGAN [ 14 ] leverages the generative adversarial network with a time-frequency masking function for singing voice separation.…”

“…Qian et al used stripe-transformer blocks to learn the deep stripe feature in encoder and decoder blocks, which are composed of residual CNN blocks [ 5 ]. Geng et al developed a gated nested U-Net(GNU-Net) architecture to generate full-resolution feature maps [ 10 ]. Yuan et al used genetic algorithms to search the effective MRP-CNN structures, which are composed of various-sized pooling operators, to extract multiresolution features.…”

“…Most DNN-based singing voice separation often consists of three stages [10,12,14] • Time-frequency transformation. The time domain signals of the singing voice and mixture are decomposed into two-dimensional time-frequency-domain spectrograms by STFT.…”

“…Researchers have proposed many deep learning models for BSS, including recurrent neural networks (RNNs) [8], convolutional neural networks (CNNs) [9], U-Nets [4,10,11], long short-term memory (LSTM) [12], generative adversarial networks (GANs) [13,14], etc. The results show that a DNN model trained with the singing voices of dozens of singers can separate the singing voices of others.…”

3 Directional Inception-ResUNet: deep spatial feature learning for multichannel singing voice separation with distortion

“…The researchers have proposed many deep learning models for BSS including recurrent neural networks(RNN) [4] [5] [11], convolutional neural networks(CNN) [12] [13], U-Net [6] [14] [15], long short-term memory(LSTM) [16] [17], generative adversarial networks (GAN) [18] [19] [20], etc. The results show that a DNN model trained with the singing voices of dozens of singers can separate the singing voices of others.…”