Blind and Neural Network-Guided Convolutional Beamformer for Joint Denoising, Dereverberation, and Source Separation

Abstract

This paper proposes an approach for optimizing a Convolutional BeamFormer (CBF) that can jointly perform denoising (DN), dereverberation (DR), and source separation (SS). First, we develop a blind CBF optimization algorithm that requires no prior information on the sources or the room acoustics, by extending a conventional joint DR and SS method. For making the optimization computationally tractable, we incorporate two techniques into the approach: the Source-Wise Factorization (SW-Fact) of a CBF and the Independent Vector Extraction (IVE). To further improve the performance, we develop a method that integrates a neural network (NN) based source power spectra estimation with CBF optimization by an inverse-Gamma prior. Experiments using noisy reverberant mixtures reveal that our proposed method with both blind and NNguided scenarios greatly outperforms the conventional state-of-theart NN-supported mask-based CBF in terms of the improvement in automatic speech recognition and signal distortion reduction performance.

Audio-Visual Multi-Channel Speech Separation, Dereverberation and Recognition

“…Possible integration solutions include: a) a pipelined architecture within which the speech separation and dereverberation components are sequentially connected in any order such as the previous researches in [21], [48], [68]; or b) a single architecture where both these two enhancement functions are implemented, for example, using weighted power minimization distortionless response (WPD) [69]- [71] and the related DNN TF-mask based WPD [72], [73] approaches. To date, such integration problem has only been investigated for audio-only speech enhancement [21], [69]- [77], but has not been studied for audio-visual speech separation and dereverberation.…”