Towards speech enhancement using a variational U-Net architecture

Abstract

In this paper, we investigate the viability of a variational U-Net architecture for denoising of single-channel audio data. Deep network speech enhancement systems commonly aim to estimate filter masks, or opt to skip preprocessing steps to directly work on the waveform signal, potentially neglecting relationships across higher dimensional spectro-temporal features. We study the adoption of a probabilistic bottleneck, as well as dilated convolutions, into the classic U-Net architecture. Evaluation of a number of network variants is carried out using signal-to-distortion ratio and perceptual model scores, with audio data including known and unknown noise types as well as reverberation. Our experiments show that the residual (skip) connections in the proposed system are required for successful end-to-end signal enhancement, i.e., without filter mask estimation. Further, they indicate a slight advantage of the variational U-Net architecture over its non-variational version in terms of signal enhancement performance under reverberant conditions. Specifically, PESQ scores show increases of 0.28 and 0.49 in reverberant and non-reverberant scenes, respectively. Anecdotal evidence points to improved suppression of impulsive noise sources with the variational end-to-end U-Net compared to the recurrent mask estimation network baseline.

A Survey of Audio Enhancement Algorithms for Music, Speech, Bioacoustics, Biomedical, Industrial, and Environmental Sounds by Image U-Net

“…As found in many audio enhancement applications the vanilla architecture usually does not generate an impressive audio quality (although comparable or slightly better than the machine learning algorithms e.g. [66], [67], [68], [69], [70], and [71]) in the output. Therefore either additional layers are added, skip connections are modified, hyper parameters are adjusted or completely new architectures are proposed in many recent AE models to improve the performance.…”

“…In contrast to the deterministic characteristics of vanilla U-Net, the VAE in the bottleneck offers increased robustness towards out-of-distribution effects, such as reverberation and unknown noise types [67].…”

“…The standard U-Net does not respond well to the train/test mismatch acoustic conditions. Implementing probabilistic bottleneck instead of deterministic in U-Net, in the denoising and dereverberation model of [67], enables it to adhere well to the unknown noise and reverberant scenarios, than the standard U-Net. The model is trained on log-scaled STFT power spectrograms.The use of symmetric filters in U-Net makes sense for regular images, as there is no difference between their x and y-axis.…”

Single-Channel Speech Enhancement with Deep Complex U-Networks and Probabilistic Latent Space Models

“…The U-Network structure is similar to an autoencoder, yet the use of probabilistic latent space models similar to variational autoencoders, was previously only used for image segmentation tasks [15]. In our previous work [16], we showed that including a probabilistic latent space model in a U-Network increases its generalization ability by introducing noise to the latent space, thereby indirectly increasing the processed feature variety of the network during training. Here, we extend our previous work to model the latent space of a complex-valued U-Network by introducing two separate latent spaces for real and imaginary part, respectively.…”