Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation

Abstract

Models for audio source separation usually operate on the magnitude spectrum, which ignores phase information and makes separation performance dependant on hyperparameters for the spectral front-end. Therefore, we investigate end-to-end source separation in the time-domain, which allows modelling phase information and avoids fixed spectral transformations. Due to high sampling rates for audio, employing a long temporal input context on the sample level is difficult, but required for high quality separation results because of long-range temporal correlations. In this context, we propose the Wave-U-Net, an adaptation of the U-Net to the one-dimensional time domain, which repeatedly resamples feature maps to compute and combine features at different time scales. We introduce further architectural improvements, including an output layer that enforces source additivity, an upsampling technique and a context-aware prediction framework to reduce output artifacts. Experiments for singing voice separation indicate that our architecture yields a performance comparable to a stateof-the-art spectrogram-based U-Net architecture, given the same data. Finally, we reveal a problem with outliers in the currently used SDR evaluation metrics and suggest reporting rank-based statistics to alleviate this problem.

Joint magnitude estimation and phase recovery using Cycle-in-Cycle GAN for non-parallel speech enhancement

“…As presented in Table 2, our proposed methods are compared with other state-of-the-art (SOTA) SE approaches under standard paired data, including GAN-based methods (e.g., SEGAN [6], MM-SEGAN [9], RSGAN [7], MetricGAN [8], SASEGAN [12] and CRGAN [11]) and Non-GAN based methods(i.e., Wave-U-net [28], DFL-SE [29], CRN-MSE [30], GCRN [31], DCCRN [32] and TF-SNN [33]). Note that we reimplement GCRN and DCCRN on VoiceBank + Demand dataset, and directly use the reported scores of other methods in their original papers.…”

DPCRN: Dual-Path Convolution Recurrent Network for Single Channel Speech Enhancement

“…As a data-driven supervised learning approach, DNN-based speech enhancement can be mainly categorized into timefrequency domain [2][3][4] and time domain [5][6][7] methods. The time-frequency (T-F) domain methods aim to extract the acoustic features (e.g., complex spectrum or logarithmic power spectrum) of clean speech from the features of noisy speech.…”

“…As a typical method in the time domain, Conv-Tasnet [5] utilizes a 1-D convolution neural network (Conv-1D) [14] as an encoder to convert time-domain waveform into effective representations for effective clean speech estimation, and then converts the representations back to waveform by a transposed convolutional layer called decoder. Time domain methods suffer from the difficulty of modeling extremely long sequences so that very deep convolutional layers like wave-u-net [7] have to be utilized for feature compression. Conventional recurrent neural networks (RNNs) are also not effective for modeling such long sequences.…”

Singer separation for karaoke content generation

“…Compared with a single-channel audio signal, a multi-channel audio signal obtains more spatial information, thereby further assisting speech separation. Wave-U-Net [2] splices multi-channel signals are input into U-Net and need to change the number of input channels, but the input length of the time domain is usually not fixed if the series is very long, and its resistance to optimization means the traditional RNN model cannot be effectively used. Dual-path recurrent neural networks (DPRNN) optimize RNN in the deep model to process extremely long speech sequences [3].…”