Depthwise-STFT Based Separable Convolutional Neural Networks

Kumawat, Sudhakar; Raman, Shanmuganathan

doi:10.1109/icassp40776.2020.9053898

Cited by 7 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that the number of frequency variables are reduced accordingly. For example, ST-STFT with K = 13 in Figure 3 uses four unique frequency variables in the spatial dimensions (i.e., [63].…”

Section: Variations Of Stft Blocksmentioning

confidence: 99%

Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

Kumawat,

Verma,

Nakashima

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Conventional 3D convolutional neural networks (CNNs) are computationally expensive, memory intensive, prone to overfitting, and most importantly, there is a need to improve their feature learning capabilities. To address these issues, we propose spatio-temporal short term Fourier transform (STFT) blocks, a new class of convolutional blocks that can serve as an alternative to the 3D convolutional layer and its variants in 3D CNNs. An STFT block consists of non-trainable convolution layers that capture spatially and/or temporally local Fourier information using a STFT kernel at multiple low frequency points, followed by a set of trainable linear weights for learning channel correlations. The STFT blocks significantly reduce the space-time complexity in 3D CNNs. In general, they use 3.5 to 4.5 times less parameters and 1.5 to 1.8 times less computational costs when compared to the state-of-the-art methods. Furthermore, their feature learning capabilities are significantly better than the conventional 3D convolutional layer and its variants. Our extensive evaluation on seven action recognition datasets, including Something 2 v1 and v2, Jester, Diving-48, Kinetics-400, UCF 101, and HMDB 51, demonstrate that STFT blocks based 3D CNNs achieve on par or even better performance compared to the state-of-the-art methods.

show abstract

Section: Variations Of Stft Blocksmentioning

confidence: 99%

Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

Kumawat,

Verma,

Nakashima

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Most of deep learning architectures for speech enhancement are formulated in the full-band time-frequency (T-F) representation of the speech mixture (Tan and Wang 2021;Hu et al 2020;Zhao and Wang 2020). By using short-time Fourier transform (STFT), the state-of-the-art methods estimate the spectrogram of the desired speech signal from the mixture spectrogram (Kumawat and Raman 2020) (Pandey and Wang 2020). However, it has been confirmed that the background noise is uniformly distributed at the full band and human speech occupies in the lower frequency-band (Li, Sun, and Naqvi 2021).…”

Section: Introductionmentioning

confidence: 99%

U-shaped Transformer with Frequency-Band Aware Attention for Speech Enhancement

Li¹,

Sun²,

Naqvi³

2021

Preprint

View full text Add to dashboard Cite

The state-of-the-art speech enhancement has limited performance in speech estimation accuracy. Recently, in deep learning, the Transformer shows the potential to exploit the longrange dependency in speech by self-attention. Therefore, it is introduced in speech enhancement to improve the speech estimation accuracy from a noise mixture. However, to address the computational cost issue in Transformer with selfattention, the axial attention is the option i.e., to split a 2D attention into two 1D attentions. Inspired by the axial attention, in the proposed method we calculate the attention map along both time-and frequency-axis to generate time and frequency sub-attention maps. Moreover, different from the axial attention, the proposed method provides two parallel multi-head attentions for time-and frequency-axis. Furthermore, it is proven in the literature that the lower frequencyband in speech, generally, contains more desired information than the higher frequency-band, in a noise mixture. Therefore, the frequency-band aware attention is proposed i.e., high frequency-band attention (HFA), and low frequency-band attention (LFA). The U-shaped Transformer is also first time introduced in the proposed method to further improve the speech estimation accuracy. The extensive evaluations over four public datasets, confirm the efficacy of the proposed method.

show abstract

“…They then updated their model to QuartzNet [1], replacing traditional 1D-convolution layers by 1D time-channel separable convolutional layers. Time-depth separable convolutions [8,9,10] are designed to reduce the number of parameters in traditional convolutions while keeping the receptive field large. The original QuartzNet model has K × C + C 2 parameters, where K is the kernel size and C is the channel dimension.…”

Section: Introductionmentioning

confidence: 99%

Multi-QuartzNet: Multi-Resolution Convolution for Speech Recognition with Multi-Layer Feature Fusion

Luo

Wang

Cheng

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, we propose an end-to-end speech recognition network based on Nvidia's previous QuartzNet [1] model. We try to promote the model performance, and design three components: (1) Multi-Resolution Convolution Module, replaces the original 1D time-channel separable convolution with multi-stream convolutions. Each stream has a unique dilated stride on convolutional operations. (2) Channel-Wise Attention Module, calculates the attention weight of each convolutional stream by spatial channel-wise pooling. (3) Multi-Layer Feature Fusion Module, reweights each convolutional block by global multi-layer feature maps. Our experiments demonstrate that Multi-QuartzNet model achieves CER 6.77% on AISHELL-1 data set, which outperforms original QuartzNet and is close to state-of-art result.

show abstract

Depthwise-STFT Based Separable Convolutional Neural Networks

Cited by 7 publications

References 15 publications

Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

U-shaped Transformer with Frequency-Band Aware Attention for Speech Enhancement

Multi-QuartzNet: Multi-Resolution Convolution for Speech Recognition with Multi-Layer Feature Fusion

Contact Info

Product

Resources

About