Self-Attention Generative Adversarial Network for Speech Enhancement

Abstract

Existing generative adversarial networks (GANs) for speech enhancement solely rely on the convolution operation, which may obscure temporal dependencies across the sequence input. To remedy this issue, we propose a self-attention layer adapted from non-local attention, coupled with the convolutional and deconvolutional layers of a speech enhancement GAN (SEGAN) using raw signal input. Further, we empirically study the effect of placing the self-attention layer at the (de)convolutional layers with varying layer indices as well as at all of them when memory allows. Our experiments show that introducing self-attention to SEGAN leads to consistent improvement across the objective evaluation metrics of enhancement performance. Furthermore, applying at different (de)convolutional layers does not significantly alter performance, suggesting that it can be conveniently applied at the highest-level (de)convolutional layer with the smallest memory overhead 1 .

Speech Enhancement Using U-Net with Compressed Sensing

“…Although the parameters of the classic Attention-Wave-U-Net [14] and the causal DEMUCS [15] are less than that of SECS-U-Net, the scores of metrics in Tables 3 and 4 show that their overall enhancement performance is inferior to SECS-U-Net. Furthermore, it also indicates that the parameters of our model only account for 29.99% of that of the recently proposed SASEGAN [7]. In addition, compared with the latest Sinc-SEGAN [9], the parameters of SECS-U-Net are 32.99% of its.…”

“…Table 3 shows the scores of speech quality metrics in the SECS-U-Net and the reference models on the open test set [30]. In Table 3, we reproduced Wiener [3], SEGAN [4], and SASEGAN [7]. The scores of other models from their papers.…”

“…It focuses on speech feature information and suppresses Part of U-Net-based speech enhancement models rely on ordinary convolution operations in the codec [4][5][6]8,15], the contextual information of the speech may be ignored and detailed features may be lost. To tackle these problems, a self-attention mechanism was added to the U-Net based generator to pay attention to the non-local detail features of speech [7]. Although the enhancement performance of this model was improved, the structure of attention was more complicated, which increased the complexities of the model.…”

“…However, multiple generators increased the model complexity and the number of parameters. In [7], a self-attention mechanism was proposed and coupled with the (de)convolutional layers of generator (SASEGAN). It paid attention to the temporal contexts of the speech.…”

A Novel Temporal Attentive-Pooling based Convolutional Recurrent Architecture for Acoustic Signal Enhancement

“…In [48], the authors investigated a multi-head attention network to estimate linear prediction coefficients (LPC) for clean and noisy speech signals. In [49][50][51][52], the authors employed convolutional neural network (CNN)-and generative adversarial network (GAN)-based architectures with attention mechanisms for end-to-end ASE applications.…”