Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition

Zhang, Shiliang; Gao, Zhifu; Luo, Haoneng; Lei, Ming; Gao, Jie; Yan, Zhijie; Xie, Lei

doi:10.48550/arxiv.2006.01712

Cited by 6 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A far-field set of about 15 hours data and a common set of about 30 hours data are used to evaluated the performance. Other configurations could be found in [21,28,29]. Real time factor (RTF) was used to measure the inference speed on GPU (NVIDIA Tesla V100).…”

Section: Methodsmentioning

confidence: 99%

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition

Gao¹,

Zhang²,

McLoughlin³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

Transformers have recently dominated the ASR field. Although able to yield good performance, they involve an autoregressive (AR) decoder to generate tokens one by one, which is computationally inefficient. To speed up inference, non-autoregressive (NAR) methods, e.g. single-step NAR, were designed, to enable parallel generation. However, due to an independence assumption within the output tokens, performance of single-step NAR is inferior to that of AR models, especially with a largescale corpus. There are two challenges to improving singlestep NAR: Firstly to accurately predict the number of output tokens and extract hidden variables; secondly, to enhance modeling of interdependence between output tokens. To tackle both challenges, we propose a fast and accurate parallel transformer, termed Paraformer. This utilizes a continuous integrate-andfire based predictor to predict the number of tokens and generate hidden variables. A glancing language model (GLM) sampler then generates semantic embeddings to enhance the NAR decoder's ability to model context interdependence. Finally, we design a strategy to generate negative samples for minimum word error rate training to further improve performance. Experiments using the public AISHELL-1, AISHELL-2 benchmark, and an industrial-level 20,000 hour task demonstrate that the proposed Paraformer can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup. 1

show abstract

Section: Methodsmentioning

confidence: 99%

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition

Gao¹,

Zhang²,

McLoughlin³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

show abstract

“…The recent years have seen the rapid development of endto-end (E2E) models applied in automatic speech recognition (ASR) field, including connection temporal classification (CTC) [8], recurrent neural network transducer (RNN-T) [9,10] and attention based encoder decoder (AED) [11,12,13], etc. Benefiting from the strong ability of multi-headed-attention (MHA) in global context modeling, transformer [11] is competitive among these models in non-streaming tasks, hence kinds of variants come out [14,15,16] and become the mainstream of ASR research.…”

Section: Introductionmentioning

confidence: 99%

Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR

Shi¹,

Zhou²,

Chen³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Neural architecture search (NAS) has been successfully applied to tasks like image classification and language modeling for finding efficient high-performance network architectures. In ASR field especially end-to-end ASR, the related research is still in its infancy. In this work, we focus on applying NAS on the most popular manually designed model: Conformer, and then propose an efficient ASR model searching method that benefits from the natural advantage of differentiable architecture search (Darts) in reducing computational overheads. We fuse Darts mutator and Conformer blocks to form a complete search space, within which a modified architecture called Darts-Conformer cell is found automatically. The entire searching process on AISHELL-1 dataset costs only 0.7 GPU days. Replacing the Conformer encoder by stacking searched cell, we get an end-to-end ASR model (named as Darts-Conformner) that outperforms the Conformer baseline by 4.7% on the opensource AISHELL-1 dataset. Besides, we verify the transferability of the architecture searched on a small dataset to a larger 2k-hour dataset. To the best of our knowledge, this is the first successful attempt to apply gradient-based architecture search in the attention-based encoder-decoder ASR model.

show abstract

“…The encoder transforms raw acoustic features into a high-level representation, while the decoder predicts output symbols in an auto-regressive manner. The Transformer-based models have dominated the seq2seq modeling in the ASR field, due to its superiority of recognition accuracy [7][8][9][10][11][12][13][14]. However, large amount model parameters become the main challenge for deploying these ASR models in resource constrained scenarios, where both memory and computation resources are limited.…”

Section: Introductionmentioning

confidence: 99%

Extremely Low Footprint End-to-End ASR System for Smart Device

Gao¹,

Yao²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, end-to-end (E2E) speech recognition has become popular, since it can integrate the acoustic, pronunciation and language models into a single neural network, as well as outperforms conventional models. Among E2E approaches, attentionbased models, e.g. Transformer, have emerged as being superior. The E2E models have opened the door of deployment of ASR on smart device, however it still suffers from large amount model parameters. This work proposes an extremely low footprint E2E ASR system for smart device, to achieve the goal of satisfying resource constraints without sacrificing recognition accuracy. We adopt cross-layer weight sharing to improve parameter-efficiency. We further exploit the model compression methods including sparsification and quantization, to reduce the memory storage and boost the decoding efficiency on smart device. We have evaluated our approach on the public AISHELL-1 and AISHELL-2 benchmarks. On the AISHELL-2 task, the proposed method achieves more than 10× compression (model size from 248MB to 24MB) while shuffer from small performance loss (CER from 6.49% to 6.92%).

show abstract

Streaming Chunk-Aware Multihead Attention for Online End-to-End Speech Recognition

Cited by 6 publications

References 19 publications

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition

Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR

Extremely Low Footprint End-to-End ASR System for Smart Device

Contact Info

Product

Resources

About