Time-Domain Audio Source Separation Based on Wave-U-Net Combined with Discrete Wavelet Transform

Abstract

We propose a time-domain audio source separation method using down-sampling (DS) and up-sampling (US) layers based on a discrete wavelet transform (DWT). The proposed method is based on one of the state-of-the-art deep neural networks, Wave-U-Net, which successively down-samples and up-samples feature maps. We find that this architecture resembles that of multiresolution analysis, and reveal that the DS layers of Wave-U-Net cause aliasing and may discard information useful for the separation. Although the effects of these problems may be reduced by training, to achieve a more reliable source separation method, we should design DS layers capable of overcoming the problems. With this belief, focusing on the fact that the DWT has an anti-aliasing filter and the perfect reconstruction property, we design the proposed layers. Experiments on music source separation show the efficacy of the proposed method and the importance of simultaneously considering the anti-aliasing filters and the perfect reconstruction property.

Upsampling layers for music source separation

“…Hence, no additive (tonal) or substractive (filtering) artifacts emerge after downsampling/upsampling with wavelets. However, the perfect reconstruction property only holds in absence of any processing in the wavelet domain-and we describe waveletinspired downsampling/upsampling layers that, combined with neural networks, have the capacity to perform downstream tasks [8,16]: -Lazy wavelet layers downsample the signal into odd/even samples, and interleave odd/even samples for upsampling. -Haar wavelet layers can be described following the filter bank scheme in Fig.…”

“…Further, wavelets provide a principled way to downsample. Inspired by that, Nakamura and Saruwatari [8] replaced WaveUnet's [2] downsampling (discarding every other time step) and upsampling (linear interpolation) layers by lazy and Haar wavelet layers. However, the above wavelet layers are designed to downsample/upsample x2, as the original WaveUnet.…”

“…We also experiment with learnable wavelet layers. To do so, we implement the above wavelets (both analysis and synthesis) using the lifting scheme [8,16] that defines wavelet transforms with 3 parameters: a prediction operator P , an update operator U , and a normalization constant A [8,16]. Haar and lazy wavelets can be implemented using the lifting scheme by setting P =1, U =0.5, A= √ 2; and P =0, U =0, A=1 (respectively) [8,16].…”

“…Filtering artifacts attenuate some bands and are known to "de-emphasize high-end frequencies" [3], while tonal artifacts introduce additive periodic noise percieved as a "high-frequency buzzing noise" [2] that "can be devastating to audio generation results" [1]. Since sorting out such artifacts is instrumental for the development of high-fidelity neural audio synthesizers, our work aims at investigating current upsampling layers to advance our understanding of those: transposed convolutions [1,5,6], interpolation upsamplers [2,7], subpixel convolutions [4], and wavelet-based upsamplers [8]. Transposed and subpixel convolutions can introduce tonal artifacts [1,2,5], whereas interpolation and wavelet-based upsamplers can produce filtering artifacts [3].…”

Time-Domain Audio Source Separation With Neural Networks Based on Multiresolution Analysis

“…Note that this paper is partially based on our international conference papers [24], [25]. This paper has the following seven additional contributions: (i) Although the DWT layer we presented in the conference paper is designed only for predetermined wavelets, we extend it to those jointly trainable with the other DNN modules.…”