Extremely Low Footprint End-to-End ASR System for Smart Device

Abstract

Recently, end-to-end (E2E) speech recognition has become popular, since it can integrate the acoustic, pronunciation and language models into a single neural network, as well as outperforms conventional models. Among E2E approaches, attentionbased models, e.g. Transformer, have emerged as being superior. The E2E models have opened the door of deployment of ASR on smart device, however it still suffers from large amount model parameters. This work proposes an extremely low footprint E2E ASR system for smart device, to achieve the goal of satisfying resource constraints without sacrificing recognition accuracy. We adopt cross-layer weight sharing to improve parameter-efficiency. We further exploit the model compression methods including sparsification and quantization, to reduce the memory storage and boost the decoding efficiency on smart device. We have evaluated our approach on the public AISHELL-1 and AISHELL-2 benchmarks. On the AISHELL-2 task, the proposed method achieves more than 10× compression (model size from 248MB to 24MB) while shuffer from small performance loss (CER from 6.49% to 6.92%).

Beyond Universal Transformer: block reusing with adaptor in Transformer for automatic speech recognit

“…It will make this block deteriorate and lose non-linear modeling called linear deterioration. This discovery could explain the model degradation in the [10,11,17] as well.…”

“…Unfortunately, its CER also almost keeps same. But compared with the similar setup Exp6 (M = N = 1) in [17], our proposed method "BRA-E" almost keeps the similar CER but just 65.4% the number of parameters in model.…”

“…Unfortunately, gradient clip experiment in "BRA-E" and "BRA-ED" don't evident improvement in this noiseless dataset. Inspired by [17,11], the S1 also is increase from 12 to 18 to explore the further improvement. Unfortunately, its CER also almost keeps same.…”

“…Based on simple block reusing, [11] also explore only block reusing, it shows that if only one share attention, the performance will drop a little, but only decrease the number of parameters by around one-third. And then [17] explore the possibility migrate its application field from language to speech. It should be noted that [18] also points out that the transformer encoder blocks have high similarity.…”

“…The WER of LR/LRA model in AISHELL test * The CG, CP, ATT, ATT-RE respectively means the decoding method: CTC greedy, CTC prefix, attention, attention resore * * the EXP6 result in the Table2in[17], similar setup with all BR/BRA experiments…”