On Loss Functions and Recurrency Training for GAN-Based Speech Enhancement Systems

Abstract

Recent work has shown that it is feasible to use generative adversarial networks (GANs) for speech enhancement, however, these approaches have not been compared to state-of-the-art (SOTA) non GAN-based approaches. Additionally, many loss functions have been proposed for GAN-based approaches, but they have not been adequately compared. In this study, we propose novel convolutional recurrent GAN (CRGAN) architectures for speech enhancement. Multiple loss functions are adopted to enable direct comparisons to other GAN-based systems. The benefits of including recurrent layers are also explored. Our results show that the proposed CRGAN model outperforms the SOTA GAN-based models using the same loss functions and it outperforms other non-GAN based systems, indicating the benefits of using a GAN for speech enhancement. Overall, the CRGAN model that combines an objective metric loss function with the mean squared error (MSE) provides the best performance over comparison approaches across many evaluation metrics.

Self-Attention Generative Adversarial Network for Speech Enhancement

“…SEGAN Given a noise-corrupted raw audio signalx = x + n ∈ R T , where x ∈ R T denotes a clean signal and n ∈ R T denotes additive background noise, the goal of speech enhancement is to find a mapping f (x) :x → x to recover the clean signal x from the noisy signalx. SEGAN methods [13,14,15] achieve this goal by designating the generator G as the enhancement mapping, i.e.x = G(z,x) where z is a latent variable. The discriminator D is tasked to distinguish the enhanced outputx from the real clean signal x.…”

“…1. Various losses have been proposed to improve adversarial training, such as least-squares loss [13,23], Wasserstein loss [14], relativistic loss [16], and metric loss, [14]. Here, we employ the least-squares loss as in the seminal work [13].…”

“…Recently, the research community has witnessed a shift in methodology from conventional signal processing methods [6,7] to data-driven enhancement approaches, particularly those based on deep learning paradigms [8,9,3,10,11]. Beside discriminative modeling with typical deep network variants, such as deep neural networks (DNNs) [8], convolutional neural networks (CNNs) [9,10], and recurrent neural networks (RNNs) [11,3], generative modeling with GANs [12] have been shown to hold promise for speech enhancement [13,14,15]. Furthermore, the study in [15] indicates that generative modeling with GANs may result in fewer artefacts than discriminative methods.…”

“…raw waveform [15,16] and time-frequency image [9,14]. Better losses, like Wasserstein loss [14], relativistic loss [16], and metric loss [14], have been tailored to gain stabilization in the training process. SEGANs that learn multi-stage enhancement mappings have also been proposed [15].…”

“…On the one hand, self-attention has been successfully used for sequential modeling in different speech modeling tasks [18,19,20]. On the other hand, it is more flexible in modeling both long-range and local dependencies and is more efficient than RNN [14] in terms of computational cost, especially when applied to long sequences. The reason is that RNN is based on temporal iterations which cannot be parallelized whereas self-attention is based on matrix multiplication which is highly parallelizable and easily accelerated.…”