End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamforming

Abstract

Despite successful applications of end-to-end approaches in multi-channel speech recognition, the performance still degrades severely when the speech is corrupted by reverberation. In this paper, we integrate the dereverberation module into the end-to-end multi-channel speech recognition system and explore two different frontend architectures. First, a multisource mask-based weighted prediction error (WPE) module is incorporated in the frontend for dereverberation. Second, another novel frontend architecture is proposed, which extends the weighted power minimization distortionless response (WPD) convolutional beamformer to perform simultaneous separation and dereverberation. We derive a new formulation from the original WPD, which can handle multi-source input, and replace eigenvalue decomposition with the matrix inverse operation to make the back-propagation algorithm more stable. The above two architectures are optimized in a fully end-toend manner, only using the speech recognition criterion. Experiments on both spatialized wsj1-2mix corpus and REVERB show that our proposed model outperformed the conventional methods in reverberant scenarios.

End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend

“…The ASR backend is a joint connectionist temporal classification (CTC) / attention-based encoder-decoder [13] model for recognizing the separated singlechannel speech. Compared to those in our previous work [22], the proposed architecture can support different beamformer variants in a single framework, by using a single mask estimator for WPE / beamforming and applying single-source WPE for processing speech of different sources.…”

“…The numerical problem generally originates from the complex operations in the WPE and beamforming formulas, such as the complex matrix inverse, leading to poor performance in certain frequency bins sparsely populated. Such behaviors are particularly undesirable in the joint training with ASR, as they can easily result in not-a-number (NaN) gradients that fail to backpropagate correctly and even prevent the model from converging properly [22], thus badly impacting the overall model performance. In order to mitigate this problem, we propose four approaches to improve the stability of both WPE and beamforming submodules:…”

“…(3) More stable complex matrix operations Due to the lack of complex support in PyTorch, the alternative method in Section 4.3 in [35] was used in our previous work [22], which tries to find a factor to construct an invertible real matrix and maps the complex inversion to some real matrix operations. But it sometimes fails due to the poor estimate of the factor that results in a singular matrix.…”

“…With the above proposed techniques, we are now able to optimize the convolutional beamformer and ASR jointly, without the need of pretraining as in [22].…”

Blind and Neural Network-Guided Convolutional Beamformer for Joint Denoising, Dereverberation, and Source Separation

“…Many useful techniques have been proposed to estimate masks, e.g., by neural networks (NNs) [3,4] and clustering microphone array signals [5,6]. The mask-based BF approach effectively optimizes BFs and Convolutional BFs (CBFs) that can jointly perform denoising (DN), dereverberation (DR), and source separation (SS) [7,8]. A drawback of this approach, however, is that ATFs and BFs are estimated based on different criteria, and thus the estimated ATFs are not guaranteed to be optimal for BF/CBF estimation.…”