DF-Conformer: Integrated architecture of Conv-TasNet and Conformer using linear complexity self-attention for speech enhancement

Abstract

Single-channel speech enhancement (SE) is an important task in speech processing. A widely used framework combines an analysis/synthesis filterbank with a mask prediction network, such as the Conv-TasNet architecture. In such systems, the denoising performance and computational efficiency are mainly affected by the structure of the mask prediction network. In this study, we aim to improve the sequential modeling ability of Conv-TasNet architectures by integrating Conformer layers into a new mask prediction network. To make the model computationally feasible, we extend the Conformer using linear complexity attention and stacked 1-D dilated depthwise convolution layers. We trained the model on 3,396 hours of noisy speech data, and show that (i) the use of linear complexity attention avoids high computational complexity, and (ii) our model achieves higher scale-invariant signal-to-noise ratio than the improved time-dilated convolution network (TDCN++), an extended version of Conv-TasNet.

MT3: Multi-Task Multitrack Music Transcription

“…Transformers have also been applied to some audio modeling tasks. For example, Transformerbased models have been used for audio classification (Gong et al, 2021;Verma & Berger, 2021), captioning (Mei et al, 2021), compression (Dieleman et al, 2021, speech recognition (Gulati et al, 2020), speaker separation (Subakan et al, 2021), and enhancement (Koizumi et al, 2021). Transformers have also been used for generative audio models Verma & Chafe, 2021), which in turn have enabled further tasks in music understanding (Castellon et al, 2021).…”