Insights Into Deep Non-Linear Filters for Improved Multi-Channel Speech Enhancement

Abstract

In a multi-channel separation task with multiple speakers, we aim to recover all individual speech signals from the mixture. In contrast to single-channel approaches, which rely on the different spectro-temporal characteristics of the speech signals, multi-channel approaches should additionally utilize the different spatial locations of the sources for a more powerful separation especially when the number of sources increases. To enhance the spatial processing in a multi-channel source separation scenario, in this work, we propose a deep neural network (DNN) based spatially selective filter (SSF) that can be spatially steered to extract the speaker of interest by initializing a recurrent neural network layer with the target direction. We compare the proposed SSF with a common endto-end direct separation (DS) approach trained using utterancewise permutation invariant training (PIT), which only implicitly learns to perform spatial filtering. We show that the SSF has a clear advantage over a DS approach with the same underlying network architecture when there are more than two speakers in the mixture, which can be attributed to a better use of the spatial information. Furthermore, we find that the SSF generalizes much better to additional noise sources that were not seen during training.

McNet: Fuse Multiple Cues for Multichannel Speech Enhancement

“…(4) Narrow-band Net [11] is one of our previous works that uses two layers of LSTM to only exploit the narrow-band spatial information (like the second module of the proposed network). ( 5) FT-JNF [14]: Kristina et al revised the Narrow-band Net [11] by replacing the first LSTM with an along-frequency LSTM to further exploit the fullband information (like the first and second modules together of the proposed network).…”

“…The first two modules together follow a similar spirit as the FT-JNF network [14]. The major difference is that, besides the output of module 1, we also feed the narrow-band noisy signals to the second module, as the first module may lose some narrow-band information.…”

“…The basic strategy is to exploit each type of information with a dedicated network. The single-channel FullSubNet [2] and multichannel FT-JNF network [14] have proved that this strategy is effective for fusing different types of information. Specifically, the proposed multi-cue fusion network (named McNet) cascades four modules, including a full-band spatial, narrow-band spatial, sub-band spectral and full-band spectral module.…”

“…Each module uses one layer of the LSTM network, by which the temporal dynamic of each type of information can be properly modeled. Compared with two SOTA networks, i.e., CA Dense U-net [12] and FT-JNF [14], experiments show that the proposed network achieves notably better speech enhancement performance.…”

“…In our previous works [11,13], a network is proposed to focus on the narrow-band spatial information, namely the difference of spatial correlation between speech and noise formulated in narrow-band. [14] cascades a full-band network with the narrow-band network to exploit the full-band information simultaneously, and achieves the state-of-the-art (SOTA) performance.…”