Uformer: A Unet Based Dilated Complex &amp; Real Dual-Path Conformer Network for Simultaneous Speech Enhancement and Dereverberation

Fu, Yihui; Liu, Yun; Li, Jingdong; Luo, Dawei; Lv, Shubo; Jv, Yukai; Xie, Lei

doi:10.1109/icassp43922.2022.9746020

Cited by 32 publications

(17 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The work [17] first applies the realvalued conformer to a two-stage modelling scheme. An extended work was proposed in [19] for a semi-complex model. In this work, we develop a fully complex model with a much simpler architecture.…”

Section: Complex Dual-path Conformer Blockmentioning

confidence: 99%

“…The outputs of the two decoders are weighted and summed. erate on the raw waveform of speech signals and the time-frequency (TF) domain approaches [10][11][12][13][14][15][16][17][18][19][20][21] that manipulate the speech spectrogram are proposed. Although the time-domain approaches have made some success, the TF domain approach has dominated the research trend.…”

Section: Introductionmentioning

confidence: 99%

“…Typically, most of recent studies treat the real and imaginary parts as two separated real-valued sequences and model them using realvalued networks [10][11][12][13][14][15][16][17]. However, the speech spectrogram and the complex targets are naturally complex-valued, much richer representations and more efficiently modelling could be potentially achieved by complex networks [18,19]. This is because the complex models confine with the complex multiply rules and may jointly learn the real and imaginary parts based on prior knowledge.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

D2Former: A Fully Complex Dual-Path Dual-Decoder Conformer Network using Joint Complex Masking and Complex Spectral Mapping for Monaural Speech Enhancement

Zhao¹,

Ma²

2023

Preprint

View full text Add to dashboard Cite

Monaural speech enhancement has been widely studied using real networks in the time-frequency (TF) domain. However, the input and the target are naturally complex-valued in the TF domain, a fully complex network is highly desirable for effectively learning the feature representation and modelling the sequence in the complex domain. Moreover, phase, an important factor for perceptual quality of speech, has been proved learnable together with magnitude from noisy speech using complex masking or complex spectral mapping. Many recent studies focus on either complex masking or complex spectral mapping, ignoring their performance boundaries. To address above issues, we propose a fully complex dual-path dual-decoder conformer network (D2Former) using joint complex masking and complex spectral mapping for monaural speech enhancement. In D2Former, we extend the conformer network into the complex domain and form a dual-path complex TF self-attention architecture for effectively modelling the complex-valued TF sequence. We further boost the TF feature representation in the encoder and the decoders using a dual-path learning structure by exploiting complex dilated convolutions on time dependency and complex feedforward sequential memory networks (CFSMN) for frequency recurrence. In addition, we improve the performance boundaries of complex masking and complex spectral mapping by combining the strengths of the two training targets into a joint-learning framework. As a consequence, D2Former takes fully advantages of the complex-valued operations, the dual-path processing, and the joint-training targets. Compared to the previous models, D2Former achieves state-of-theart results on the VoiceBank+Demand benchmark with the smallest model size of 0.87M parameters.

show abstract

Section: Complex Dual-path Conformer Blockmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

D2Former: A Fully Complex Dual-Path Dual-Decoder Conformer Network using Joint Complex Masking and Complex Spectral Mapping for Monaural Speech Enhancement

Zhao¹,

Ma²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In recent years, there has been a similar trend of conventional speech dereverberation approaches [24]- [27] such as WPE evolving into their current DNN based variants. These include: a) the DNN-WPE [22], [23] method, which uses neural network estimated target signal PSD matrices in place of those traditionally obtained using maximum likelihood estimation trained complex value Gaussian Mixture Models [24] in the dereverberation filter estimation; and b) complex spectral masking [28], [29] and spectral mapping [30], [31] learning a transformation between reverberant and anechoic data.…”

Section: Introductionmentioning

confidence: 99%

Audio-Visual Multi-Channel Speech Separation, Dereverberation and Recognition

Deng

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Accurate recognition of cocktail party speech containing overlapping speakers, noise and reverberation remains a highly challenging task to date. Motivated by the invariance of visual modality to acoustic signal corruption, an audio-visual multi-channel speech separation, dereverberation and recognition approach featuring a full incorporation of visual information into all system components is proposed in this paper. The efficacy of the video input is consistently demonstrated in maskbased MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end and Conformer ASR back-end. Audio-visual integrated front-end architectures performing speech separation and dereverberation in a pipelined or joint fashion via mask-based WPD are investigated. The error cost mismatch between the speech enhancement front-end and ASR back-end components is minimized by end-to-end jointly fine-tuning using either the ASR cost function alone, or its interpolation with the speech enhancement loss. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset. The proposed audio-visual multi-channel speech separation, dereverberation and recognition systems consistently outperformed the comparable audio-only baseline by 9.1% and 6.2% absolute (41.7% and 36.0% relative) word error rate (WER) reductions. Consistent speech enhancement improvements were also obtained on PESQ, STOI and SRMR scores 1 .

show abstract

“…A complex-valued U-Net was proposed [11], causing attention to shift to phase-aware networks [12]. Showing state-of-the-art performance, complexvalued approaches were further developed [13], including transformer-based U-Networks [14]. The U-Network structure is similar to an autoencoder, yet the use of probabilistic latent space models similar to variational autoencoders, was previously only used for image segmentation tasks [15].…”

mentioning

confidence: 99%

Single-Channel Speech Enhancement with Deep Complex U-Networks and Probabilistic Latent Space Models

Nustede

Anemüller

2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this paper, we propose to extend the deep, complex U-Network architecture for speech enhancement by incorporating a probabilistic (i.e., variational) latent space model. The proposed model is evaluated against several ablated versions of itself in order to study the effects of the variational latent space model, complex-value processing, and self-attention. Evaluation on the MS-DNS 2020 and Voicebank+Demand datasets yields consistently high performance. E.g., the proposed model achieves an SI-SDR of up to 20.2 dB, about 0.5 to 1.4 dB higher than its ablated version without probabilistic latent space, 2-2.4 dB higher than WaveUNet, and 6.7 dB above PHASEN. Compared to real-valued magnitude spectrogram processing with a variational U-Net, the complex U-Net achieves an improvement of up to 4.5 dB SI-SDR. Complex spectrum encoding as magnitude and phase yields best performance in anechoic conditions whereas real and imaginary part representation results in better generalization to (novel) reverberation conditions, possibly due to the underlying physics of sound.

show abstract

Uformer: A Unet Based Dilated Complex & Real Dual-Path Conformer Network for Simultaneous Speech Enhancement and Dereverberation

Cited by 32 publications

References 25 publications

D2Former: A Fully Complex Dual-Path Dual-Decoder Conformer Network using Joint Complex Masking and Complex Spectral Mapping for Monaural Speech Enhancement

D2Former: A Fully Complex Dual-Path Dual-Decoder Conformer Network using Joint Complex Masking and Complex Spectral Mapping for Monaural Speech Enhancement

Audio-Visual Multi-Channel Speech Separation, Dereverberation and Recognition

Single-Channel Speech Enhancement with Deep Complex U-Networks and Probabilistic Latent Space Models

Contact Info

Product

Resources

About