4-bit Conformer with Native Quantization Aware Training for Speech Recognition

Ding, Shaojin; Phoenix, Meadowlark,; He, You; Lew, Łukasz; Agrawal, Shivani; Rybakov, Oleg

doi:10.48550/arxiv.2203.15952

Cited by 2 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…the best checkpoint based on dev-clean WER and observe no degradation from the test sets with the model size reduced by 6.4×. Although this seems counter-intuitive in that the compressed model outperforms the 32-bit baseline, it is not rare as also shown in [11]. One explanation is that by driving weights towards quantization centroids, the search space is drastically reduced, yielding an arguably easier optimization process.…”

Section: Modelmentioning

confidence: 93%

“…In contrast, FP-QAT can be regularizer free [11,21]. Usually, the process is to use a "fake quantizer" or equivalent operations during training, hard quantizing weights to a specific range and bit-depth; and then at runtime, converting the model to INT8 format via TFLite [22].…”

Section: Related Qat Approachesmentioning

confidence: 99%

“…Usually, the process is to use a "fake quantizer" or equivalent operations during training, hard quantizing weights to a specific range and bit-depth; and then at runtime, converting the model to INT8 format via TFLite [22]. The study [11] uses native quantization operators with which, during training, the weights are quantized and then converted to the integer type for model deployment. However, FP-QAT is essentially hard compression recurring during training with severely dropped performance when applied to S8B quantization.…”

Section: Related Qat Approachesmentioning

confidence: 99%

“…PTQ is applied after the model training is complete by compressing models into 8-bit representations and is relatively well supported by various libraries [3,4,5,6,7,8], such as TensorFlow Lite [9] and AIMET [10] for on-device deployment. However, almost no existing PTQ supports customized quantization configurations to compress machine learning (ML) layers and kernels into sub-8-bit (S8B) regimes [11].…”

Section: Introductionmentioning

confidence: 99%

“…The QAT mechanism can be in the forward pass (FP-QAT) or the backward pass (BP-QAT), with the difference being whether regularization is used in the loss function. FP-QAT [11] quantizes the model weights during forward propagation to pre-defined quantization centroids. BP-QAT [1,14,15] relies on customized regularizers to gradually force weights to those quantization centroids (i.e., "soft quantization" via gradient) during training before hard compression performs in the late training phase.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

Zhang¹,

Radfar²,

Nguyen³

et al. 2022

Preprint

View full text Add to dashboard Cite

For on-device automatic speech recognition (ASR), quantization aware training (QAT) is ubiquitous to achieve the trade-off between model predictive performance and efficiency. Among existing QAT methods, one major drawback is that the quantization centroids have to be predetermined and fixed. To overcome this limitation, we introduce a regularization-free, "soft-to-hard" compression mechanism with self-adjustable centroids in a µ-Law constrained space, resulting in a simpler yet more versatile quantization scheme, called General Quantizer (GQ). We apply GQ to ASR tasks using Recurrent Neural Network Transducer (RNN-T) and Conformer architectures on both LibriSpeech and de-identified far-field datasets. Without accuracy degradation, GQ can compress both RNN-T and Conformer into sub-8-bit, and for some RNN-T layers, to 1-bit for fast and accurate inference. We observe a 30.73% memory footprint saving and 31.75% user-perceived latency reduction compared to 8-bit QAT via physical device benchmarking.

show abstract

Section: Modelmentioning

confidence: 93%

Section: Related Qat Approachesmentioning

confidence: 99%

Section: Related Qat Approachesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

Zhang¹,

Radfar²,

Nguyen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

4-bit Conformer with Native Quantization Aware Training for Speech Recognition

Ding¹,

Phoenix²,

He³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploying these massive USMs is extremely expensive due to the enormous memory usage and computational cost. Therefore, model compression is an important research topic to fit USM-based ASR under budget in real-world scenarios. In this study, we propose a USM fine-tuning approach for ASR, with a low-bit quantization and N :M structured sparsity aware paradigm on the model weights, reducing the model complexity from parameter precision and matrix topology perspectives. We conducted extensive experiments with a 2-billion parameter USM on a largescale voice search dataset to evaluate our proposed method. A series of ablation studies validate the effectiveness of up to int4 quantization and 2:4 sparsity. However, a single compression technique fails to recover the performance well under extreme setups including int2 quantization and 1:4 sparsity. By contrast, our proposed method can compress the model to have 9.4% of the size, at the cost of only 7.3% relative word error rate (WER) regressions. We also provided in-depth analyses on the results and discussions on the limitations and potential solutions, which would be valuable for future studies.

show abstract

4-bit Conformer with Native Quantization Aware Training for Speech Recognition

Cited by 2 publications

References 0 publications

Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

4-bit Conformer with Native Quantization Aware Training for Speech Recognition

Contact Info

Product

Resources

About