Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders

Abstract

Dynamical variational auto-encoders (DVAEs) are a class of deep generative models with latent variables, dedicated to time series data modeling. DVAEs can be considered as extensions of the variational autoencoder (VAE) that include the modeling of temporal dependencies between successive observed and/or latent vectors in data sequences. Previous work has shown the interest of DVAEs and their better performance over the VAE for speech signals (spectrogram) modeling. Independently, the VAE has been successfully applied to speech enhancement in noise, in an unsupervised noise-agnostic set-up that does not require the use of a parallel dataset of clean and noisy speech samples for training, but only requires clean speech signals. In this paper, we extend those works to DVAE-based single-channel unsupervised speech enhancement, hence exploiting both speech signals unsupervised representation learning and dynamics modeling. We propose an unsupervised speech enhancement algorithm based on the most general form of DVAEs, that we then adapt to three specific DVAE models to illustrate the versatility of the framework. More precisely, we combine DVAE-based speech priors with a noise model based on nonnegative matrix factorization, and we derive a variational expectation-maximization (VEM) algorithm to perform speech enhancement. Experimental results show that the proposed approach based on DVAEs outperforms its VAE counterpart and a supervised speech enhancement baseline.

Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

“…Thus, they are trained solely to generate clean speech and are therefore considered more robust to different acoustic environments compared to their discriminative counterparts. In fact, generative approaches have shown to perform better under mismatched training and test conditions [8,11,12,13]. However, they are currently less studied and still lag behind discriminative approaches, which is a strong incentive to conduct more research to realize their full potential.…”

Learning and controlling the source-filter representation of speech with a variational autoencoder

“…The interest of the present paper is rather to advance on the understanding of deep generative modeling of speech signals, while comparing honestly with highly-specialized traditional systems for which signal models are generally specifically designed for the task at hand. Moreover, advancing on the interpretability and control of the VAE latent space could be beneficial for downstream tasks, for instance to develop pitch-informed extensions of VAE-based speech enhancement methods such as those of Bando et al (2018); Leglaive et al (2018Leglaive et al ( , 2020; Bie et al (2021).…”

Speech Enhancement Based on Cyclegan with Noise-informed Training

“…Although these approaches have been proven to provide superior noise-free utterances in stationary noisy environments, they perform poorly in estimating noise with non-stationary statistics. To tackle this issue, several researchers have used deep-learning-based models to develop an unsupervised SE [23,24,25,26]. One example is Cycle-consistent GAN (CycleGAN), which was originally proposed for unpaired image-to-image translations [27] and has been successfully applied to voice conversion [28] and ASR [29].…”