FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks

Abstract

Deep dilated temporal convolutional networks (TCN) have been proved to be very effective in sequence modeling. In this paper we propose several improvements of TCN for end-to-end approach to monaural speech separation, which consists of 1) multi-scale dynamic weighted gated dilated convolutional pyramids network (FurcaPy), 2) gated TCN with intra-parallel convolutional components (FurcaPa), 3) weight-shared multi-scale gated TCN (FurcaSh), 4) dilated TCN with gated difference-convolutional component (FurcaSu), that all these networks take the mixed utterance of two speakers and maps it to two separated utterances, where each utterance contains only one speaker's voice. For the objective, we propose to train the network by directly optimizing utterance level signal-to-distortion ratio (SDR) in a permutation invariant training (PIT) style. Our experiments on the the public WSJ0-2mix data corpus results in 18.4dB SDR improvement, which shows our proposed networks can leads to performance improvement on the speaker separation task.

WHAMR!: Noisy and Reverberant Single-Channel Speech Separation

“…The resulting weights can be applied to a reconstruction set of basis functions and summed together along the same sliding window to reconstruct the signal under a similar paradigm to overlap-and-add for the STFT. For internal masking, we evaluate both bi-directional long short-term memory (BLSTM) networks (the typical internals of earlier deep learning-based speech separation systems [1][2][3][4]11,15]) and temporal convolutional networks (TCN) [16] with dilated convolutions (popular in recent state-of-the-art separation techniques [5,6]).…”

“…Great advancements have been made in recent years on solving the speech separation problem through deep learning-based techniques [1][2][3][4][5][6]. However, the overwhelming majority of research conducted thus far has used the wsj0-2mix dataset [1], which consists of synthetically-mixed studio recordings of read utterances from the WSJ0 corpus [7] and is not representative of many real-world scenarios in which overlapped speech may be present [8].…”

Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation

“…Recent progress in deep learning-based speech separation has ignited the interest of the research community in time-domain approaches [1][2][3][4][5][6]. Compared with standard time-frequency domain methods, time-domain methods are designed to jointly model the magnitude and phase information and allow direct optimization with respect to both time-and frequency-domain differentiable criteria [7][8][9].…”

“…The adaptive front-end approaches aim at replacing the short-time Fourier transform (STFT) with a differentiable transform to build a front-end that can be learned jointly with the separation network. Separation is applied to the front-end output as with the conventional time-frequency domain methods applying the separation processes to spectrogram inputs [3][4][5]. Being independent of the traditional time-frequency analysis paradigm, these systems are able to have a much more flexible choice on the window size and the number of basis functions for the front-end.…”

“…Moreover, the first RNN layer in most hierarchical RNNs still receives the entire input sequence, while in stacked DPRNN each intra-or inter-chunk RNN receives the same sublinear input size across all blocks. Compared with CNN-based architectures such as temporal convolutional networks (TCNs) that only perform local modeling due to the fixed receptive fields [4,5,20], DPRNN is able to fully utilize global information via the inter-chunk RNNs and achieve superior performance with an even smaller model size. In Section 4 we will show that by simply replacing TCN by DPRNN in a previously proposed time-domain separation system [4], the model is able to arXiv:1910.06379v1 [eess.AS] 14 Oct 2019 achieve a 0.7 dB (4.6%) relative improvement with respect to scaleinvariant signal-to-noise ratio (SI-SNR) [8] on WSJ0-2mix with a 49% smaller model size.…”