Enhancing End-to-End Multi-Channel Speech Separation Via Spatial Feature Learning

Abstract

Hand-crafted spatial features (e.g., inter-channel phase difference, IPD) play a fundamental role in recent deep learning based multi-channel speech separation (MCSS) methods. However, these manually designed spatial features are hard to incorporate into the end-to-end optimized MCSS framework. In this work, we propose an integrated architecture for learning spatial features directly from the multi-channel speech waveforms within an end-to-end speech separation framework. In this architecture, time-domain filters spanning signal channels are trained to perform adaptive spatial filtering. These filters are implemented by a 2d convolution (conv2d) layer and their parameters are optimized using a speech separation objective function in a purely data-driven fashion. Furthermore, inspired by the IPD formulation, we design a conv2d kernel to compute the inter-channel convolution differences (ICDs), which are expected to provide the spatial cues that help to distinguish the directional sources. Evaluation results on simulated multi-channel reverberant WSJ0 2-mix dataset demonstrate that our proposed ICD based MCSS model improves the overall signal-to-distortion ratio by 10.4% over the IPD based MCSS model.

Count And Separate: Incorporating Speaker Counting For Continuous Speaker Separation

“…Considerable progress has been made towards solving the talker-independent speaker separation problem, since deep clustering (DC) [1] and permutation invariant training (PIT) [2] were proposed to address the label permutation problem. To further improve separation, subsequent studies leverage microphone array processing [3]- [6], magnitude-and complex-domain phase estimation [7], [8], time-domain processing [9], and extra information such as speaker embeddings [10] and visual cues [11]. On wsj0-2mix and 3mix [1], a popular benchmark dataset containing monaural anechoic twoand three-speaker mixtures, current state-of-the-art approaches produce separation results that sound almost indistinguishable from clean speech, and the performance improvement measured by scaleinvariant signal-to-distortion ratio is more than 20 dB over no processing [12].…”

Time-Domain Speech Extraction with Spatial Information and Multi Speaker Conditioning Mechanism

“…Recently, neural network based multi-channel speech separation approaches have achieved state-of-the-art performance by directly processing time-domain speech signals [12,14]. These systems incorporate a spectral encoder, a spatial encoder, a separator, and a decoder.…”

Multi-Channel Target Speech Extraction with Channel Decorrelation and Target Speaker Adaptation

“…Actually, one of the keys of the TSE is still speech separation. In order to further enhance the separation ability, many strategies for exploiting the multi-channel information have been recently proposed, such as normalized crosscorrelation (NCC) [15], transform-average-concatenate (TAC) [16], and inter-channel convolution difference (ICD) [17], etc. Therefore, how to effectively exploit the multi-channel spatial information for TSE is crucial.…”