QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions

Kriman, Samuel; Beliaev, Stanislav; Ginsburg, Boris; Huang, Jocelyn; Kuchaiev, Oleksii; Lavrukhin, Vitaly; Leary, R. Bret; Li, Jason; Zhang, Yang

doi:10.48550/arxiv.1910.10261

Cited by 13 publications

(20 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 2 compares the (WER) result of our model on Lib-riSpeech test-clean/test-other with a few state-of-the-art models include: ContextNet [10], Transformer transducer [7], and QuartzNet [9]. All our evaluation results round up to 1 digit after decimal point.…”

Section: Results On Librispeechmentioning

confidence: 99%

“…Recently, the Transformer architecture based on self-attention [6,7] has enjoyed widespread adoption for modeling sequences due to its ability to capture long distance interactions and the high training efficiency. Alternatively, convolutions have also been successful for ASR [8,9,10,11,12], which capture local context progressively via a local receptive field layer by layer.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Conformer: Convolution-augmented Transformer for Speech Recognition

Gulati

Qin

Chiu

et al. 2020

Preprint

354

373

View full text Add to dashboard Cite

Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs). Transformer models are good at capturing content-based global interactions, while CNNs exploit local features effectively. In this work, we achieve the best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter-efficient way. To this regard, we propose the convolution-augmented transformer for speech recognition, named Conformer. Conformer significantly outperforms the previous Transformer and CNN based models achieving state-of-the-art accuracies. On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.

show abstract

Section: Results On Librispeechmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Conformer: Convolution-augmented Transformer for Speech Recognition

Gulati

Qin

Chiu

et al. 2020

Preprint

354

373

View full text Add to dashboard Cite

show abstract

“…The TDS architecture we use here also results in much lighter weight yet still low WER models. Other architectures, such as the Time-Channel Separable convolution have also shown similar gains in computational efficiency with little if any hit to accuracy [21].…”

Section: Related Workmentioning

confidence: 99%

Scaling Up Online Speech Recognition Using ConvNets

Pratap¹,

Xu²,

Kahn³

et al. 2020

Interspeech 2020

View full text Add to dashboard Cite

We design an online end-to-end speech recognition system based on Time-Depth Separable (TDS) convolutions and Connectionist Temporal Classification (CTC). We improve the core TDS architecture in order to limit the future context and hence reduce latency while maintaining accuracy. The system has almost three times the throughput of a well tuned hybrid ASR baseline while also having lower latency and a better word error rate. Also important to the efficiency of the recognizer is our highly optimized beam search decoder. To show the impact of our design choices, we analyze throughput, latency, accuracy, and discuss how these metrics can be tuned based on the user requirements.

show abstract

“…In the case of the Oracle Teacher, we trained the model for 30 epochs on a single 12GB GPU, which took about 22 hours (≈ 1 day) to finish the training. Considering that the reported training of the Quartznet [67], which is more computationally efficient than Jasper DR, for 400 epochs took 122 hours (≈ 5 days) with eight 32GB GPUs with a batch size of 256, the Oracle Teacher dramatically reduced the computational cost of the teacher model.…”

Section: B Computational Cost Comparisonmentioning

confidence: 99%

Oracle Teacher: Towards Better Knowledge Distillation

Yoon¹,

Kim²,

Lee³

et al. 2021

Preprint

View full text Add to dashboard Cite

Knowledge distillation (KD), best known as an effective method for model compression, aims at transferring the knowledge of a bigger network (teacher) to a much smaller network (student). Conventional KD methods usually employ the teacher model trained in a supervised manner, where output labels are treated only as targets. Extending this supervised scheme further, we introduce a new type of teacher model for KD, namely Oracle Teacher, that utilizes the embeddings of both the source inputs and the output labels to extract a more accurate knowledge to be transferred to the student. The proposed model follows the encoder-decoder attention structure of the Transformer network, which allows the model to attend to related information from the output labels. Extensive experiments are conducted on three different sequence learning tasks: speech recognition, scene text recognition, and machine translation. From the experimental results, we empirically show that the proposed model improves the students across these tasks while achieving a considerable speed-up in the teacher model's training time.

show abstract

QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions

Cited by 13 publications

References 12 publications

Conformer: Convolution-augmented Transformer for Speech Recognition

Conformer: Convolution-augmented Transformer for Speech Recognition

Scaling Up Online Speech Recognition Using ConvNets

Oracle Teacher: Towards Better Knowledge Distillation

Contact Info

Product

Resources

About