DeFT-AN: Dense Frequency-Time Attentive Network for Multichannel Speech Enhancement

Abstract

In this work, we present DeFTAN-II, an efficient multichannel speech enhancement model based on transformer architecture and subgroup processing. Despite the success of transformers in speech enhancement, they face challenges in capturing local relations, reducing the high computational complexity, and lowering memory usage. To address these limitations, we introduce subgroup processing in our model, combining subgroups of locally emphasized features with other subgroups containing original features. The subgroup processing is implemented in several blocks of the proposed network. In the proposed split dense blocks extracting spatial features, a pair of subgroups is sequentially concatenated and processed by convolution layers to effectively reduce the computational complexity and memory usage. For the F-and T-transformers extracting temporal and spectral relations, we introduce crossattention between subgroups to identify relationships between locally emphasized and non-emphasized features. The dualpath feedforward network then aggregates attended features in terms of the gating of local features processed by dilated convolutions. Through extensive comparisons with state-of-theart multichannel speech enhancement models, we demonstrate that DeFTAN-II with subgroup processing outperforms existing methods at significantly lower computational complexity. Moreover, we evaluate the model's generalization capability on realworld data without fine-tuning, which further demonstrates its effectiveness in practical scenarios.

Beamforming and lightweight GRU neural network combination model for multi-channel speech enhancement

“…Finally, the performance of the fusion network structure proposed in this paper is compared with commonly used methods reported in the literature, as shown in Table 3. The DeF T AN [14] multi-channel speech enhancement model utilizes a transformer structure based on multi-head attention mechanism, resulting in high computational complexity (MAC/s) . On the other hand, the FaNet [30] network structure implements a temporal convolution structure with relatively large parameter size.…”

“…Currently, most neural network-based singlechannel and multi-channel speech enhancements utilize end-to-end neural network structures. Based on their fundamental network structure, they can generally be categorized into convolutional neural network (CNN) [8], [9], [10] recurrent neural network (RNN) [11], [12], encoder-decoder structures [13], transformer structures [14] centered around attention mechanisms [7], among others. Part of speech enhancement based on deep neural network is to estimate the mask of speech time-frequency unit by using neural networks, such as ideal binary masks [15] and ideal ratio masks [16].…”

Beamforming and lightweight GRU neural networkcombination model for multi-channel speech enhancement

“…Currently, most neural network-based singlechannel and multi-channel speech enhancements utilize end-to-end neural network structures. Based on their fundamental network structure, they can generally be categorized into convolutional neural network (CNN) [8], [9], [10] recurrent neural network (RNN) [11], [12], encoder-decoder structures [13], transformer structures [14] centered around attention mechanisms [7], among others. Part of speech enhancement based on deep neural network is to estimate the mask of speech time-frequency unit by using neural networks, such as ideal binary masks [15] and ideal ratio masks [16].…”