FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement

Abstract

Convolutional recurrent networks (CRN) integrating a convolutional encoder-decoder (CED) structure and a recurrent structure have achieved promising performance for monaural speech enhancement. However, feature representation across frequency context is highly constrained due to limited receptive fields in the convolutions of CED. In this paper, we propose a convolutional recurrent encoderdecoder (CRED) structure to boost feature representation along the frequency axis. The CRED applies frequency recurrence on 3D convolutional feature maps along the frequency axis following each convolution, therefore, it is capable of catching long-range frequency correlations and enhancing feature representations of speech inputs. The proposed frequency recurrence is realized efficiently using a feedforward sequential memory network (FSMN). Besides the CRED, we insert two stacked FSMN layers between the encoder and the decoder to model further temporal dynamics. We name the proposed framework as Frequency Recurrent CRN (FRCRN). We design FRCRN to predict complex Ideal Ratio Mask (cIRM) in complex-valued domain and optimize FRCRN using both time-frequency-domain and time-domain losses. Our proposed approach achieved state-of-the-art performance on wideband benchmark datasets and achieved 2nd place for the real-time fullband track in terms of Mean Opinion Score (MOS) and Word Accuracy (WAcc) in the ICASSP 2022 Deep Noise Suppression (DNS) challenge.

D2Former: A Fully Complex Dual-Path Dual-Decoder Conformer Network using Joint Complex Masking and Complex Spectral Mapping for Monaural Speech Enhancement

“…The CFSMN module contains a CFSMN layer to learn the long-range frequency correlations. The details of the CFSMN layer are provided in our previous work [21]. Here we use the same settings for CFSMN.…”

“…The outputs of the two decoders are weighted and summed. erate on the raw waveform of speech signals and the time-frequency (TF) domain approaches [10][11][12][13][14][15][16][17][18][19][20][21] that manipulate the speech spectrogram are proposed. Although the time-domain approaches have made some success, the TF domain approach has dominated the research trend.…”

“…This is because the complex models confine with the complex multiply rules and may jointly learn the real and imaginary parts based on prior knowledge. In our previous studies, we have built complex models based on the convolutional recurrent structure and achieved promising results [20,21]. Recently, the sequence modelling capabilities have been dramatically boosted by Transformer models, especially Conformer [23].…”

Enhancement of Boring Vibrations Based on Cascaded Dual-Domain Features Extraction for Insect Pest Agrilus planipennis Monitoring

“…The CNN module is able to extract high-level features but mainly focuses on local temporal-spectral patterns [60]. Combining their advantages, the CRN structure has been shown to be very effective for speech enhancement [58,[61][62][63]. Motivated by [58,61], we determined three convolution layers for the encoder, three transposed convolution layers for the decoder, and two LSTM layers between them.…”

U-Shaped Low-Complexity Type-2 Fuzzy LSTM Neural Network for Speech Enhancement

“…With a deep structure of hidden layers between input and output layers, deep learning constructs complex models for nonlinear relations and enables feature representation from the lower layers to model the complex input data. Given a speech dataset of the clean-noisy pairs, a neural model learns to transform the noisy magnitude spectra to their clean counterparts (mapping-based SE) or estimates the time-frequency masks (masking-based SE), such as the ideal binary mask (IBM) [6], [7], ideal ratio mask (IRM) [8], [9], and spectral magnitude mask (SMM) [10]. In spectral mapping, the models are trained using a direct mapping rule, where the noisy spectral features are learned to estimate the clean spectral features.…”