Sudo RM -RF: Efficient Networks for Universal Audio Source Separation

Attention Is All You Need In Speech Separation

“…The proposed model is based on the learned-domain masking approach [14,15,[17][18][19][20][21][22] and employs an encoder, a decoder, and a masking network, as shown in Figure 1. The encoder is fully convolutional, while the masking network employs two Transformers embedded inside the dual-path processing block proposed in [17].…”

Group Communication With Context Codec for Lightweight Source Separation

“…Although the selected Sudo rm -rf configuration has a large enough receptive field to cover the entire sequential feature, it obtains an even worse separation performance with on-par model size and complexity as the TCN architecture. Although [15] reported that the Sudo rm -rf architecture achieved constantly better performance than DPRNN and TCN architectures, the results here indicates that its performance on the more challenging noisy reverberant environments needs to be revised. Moreover, although all four architectures achieve significant SI-SDR improvement with respect to the unprocessed mixture, the improvement on wideband PESQ and STOI scores are moderate.…”

“…Since the context codec squeezes the long sequence by a factor of C/2 (16 for C = 32), the effective temporal receptive field of the TCN separator is significantly larger (0.253 × 16 = 4.05s) and thus can better capture the temporal dependencies. Since it has also been reported in [15] that a deeper Sudo rm -rf architecture can lead to better overall separation performance, introducing GC3 to Sudo rm -rf might also be equivalent to increasing the model depth and improves the performance. More in-depth analysis on the reason behind the performance improvements in different architectures is left for future work.…”

“…Squeezing the input contexts into higher-level representations corresponds to a nonlinear downsampling step that generates context-level embeddings and significantly decreases the length of a feature sequence. Note that compared with other architectures that perform iterative downsampling and upsampling steps [9], [15], the context codec is only applied once and all remaining modeling steps are applied on the downsampled features, which enables a smaller memory footprint and faster training speed. We call the combination of GroupComm and Context Codec the GC3 design.…”

“…In this paper, we also explore different architectures for the GroupComm module and investigate the effect of different hyperparameters in the system configuration. Moreover, to validate the effect of GC3 on different network architectures, we select three other separation modules beyond the original dual-path RNN (DPRNN) baseline [22] applied in our previous work on GroupComm: two other CNN-based architectures, namely the temporal convolutional network (TCN) [11] and the sudo rm -rf network [15], and one transformer-based architecture, namely the dual-path transformer network (DPTNet) [24]. Experimental results show that the GC3-equipped DPRNN can achieve on-par performance with the baseline DPRNN with 4.7% model size and 17.6% MAC operations, the GC3equipped CNN-based models can significantly improve the overall performance with as few as 2.5% model size and 33.7% MAC operations, and the GC3-equipped transformer-based model can maintain on-par performance with 4.6% model size and 17.7% MAC operations.…”