Improved Relativistic Cycle-Consistent GAN With Dilated Residual Network and Multi-Attention for Speech Enhancement

Abstract

Generative adversarial networks (GANs) have been increasingly used as feature mapping functions in speech enhancement, in which the noisy speech features are transformed to the clean ones through the generators. This paper proposes a novel speech enhancement model based on a cycle-consistent relativistic GAN with Dilated Residual Networks and a Multi-attention mechanism. Using the adversarial loss, improved cycle-consistency losses, and an identity-mapping loss, a noisy-to-clean generator G and an inverse clean-to-noisy generator F simultaneously learn the forward and backward mappings between the source and target domains. To guarantee the stability of the training process, we replace vanilla GAN loss with relativistic average GAN loss and use spectral normalization in discriminators so that they conform to Lipschitz continuity. Furthermore, we employ two attention-based components as multi-attention mechanism to reduce importing signal distortion: attention U-net gates and dilated residual self-attention blocks. By employing these components, our proposed generators can capture long-term inner dependencies between elements of speech features and further preserve linguistic information. Experimental results on a public dataset indicate that the proposed model achieves state-of-the-art speech enhancement performance, especially in reducing speech distortion and improving signal overall quality. Compared with the representative GAN-based approaches, the proposed method significantly achieves the best performance in terms of STOI, CSIG, COVL, and CBAK objective metrics. Moreover, we demonstrate the contribution of each proposed component including relativistic average loss, attention U-net gate, self-attention layers, spectral normalization, and dilation operation by ten comparison systems. INDEX TERMS speech enhancement, cycle-consistent GAN, relativistic average loss, multi-attention, dilated residual network, U-net.

A Study of Spatial Attention and Squeeze Excitation Block Fusion Improved ResNet for Identifying Bank Notes

“…Traditional Optical Character Recognition (OCR) is based on image processing (minimization, texture analysis, connected domain analysis, etc.) [2][3][4]. Modern business bills are of many types and placed randomly, so it is difficult to achieve good recognition results with traditional OCR detection methods.…”

Joint magnitude estimation and phase recovery using Cycle-in-Cycle GAN for non-parallel speech enhancement

“…The AHA module integrates different hierarchical feature maps given all outputs of ATFA modules and fuse the global context of all intermediate oupputs with different weights, thus helping to guide the feature learning progressively. The output channels of each layer in encoder are (16,32,64), while the kernel size and strides are set to (3,5) and (1, 2) along the time and frequency axes, respectively.…”

“…To tackle this problem, cycle-consistent GAN (CycleGAN) has been developed to conduct unsupervised SE, which was originally used for unpaired image-to-image translation [13]. In the SE area, CycleGAN-based approaches demonstrate the remarkable ability in preserving the speech structure and improving speech quality in both paired and unpaired cases [14,15,16,17,18]. Nevertheless, these conventional CycleGAN-based approaches have several limitations for non-parallel SE.…”

Towards More Efficient DNN-Based Speech Enhancement Using Quantized Correlation Mask

“…Over the last decade, deep learning has been widely utilized for SE and has demonstrated exceptional denoising capability even in challenging conditions such as when dealing with nonstationary noise, unseen and low SNR conditions. Supervised deep neural networks (DNNs), which include feedforward multilayer perceptrons [15], convolutional neural networks [19], recurrent neural networks [20] and generative adversarial networks [21], in particular, have significantly elevated the performance of SE through capturing the complicated relationship between the noisy speech and clean speech. There has also been an increase in the use of modified versions of conventional neural networks.…”