Low-Bit Quantization and Quantization-Aware Training for Small-Footprint Keyword Spotting

Mishchenko, Yuriy; Gören, Yusuf; Sun, Ming; Beauchene, Chris; Matsoukas, Spyros; Rybakov, Oleg; Vitaladevuni, Shiv

doi:10.1109/icmla.2019.00127

Cited by 19 publications

(9 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For wakeword detection, the DNN posteriors are smoothed across the time dimension using a running average sliding window of 25 frames. In Alexa wakeword model training settings, these models provide a strong baseline performance of small footprint keyword spotting [10].…”

Section: Model Architecture and Training Parametersmentioning

confidence: 99%

Exploring the application of synthetic audio in training keyword spotters

Werchniak

Chicote

Mishchenko

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

The study of keyword spotting, a subfield within the broader field of speech recognition that centers around identifying individual keywords in speech audio, has gained particular importance in recent years with the rise of personal voice assistants such as Alexa. As voice assistants aim to rapidly expand to support new languages, keywords, and use cases, stakeholders face the issue of limited training data for these unseen scenarios. This paper details some initial exploration into the application of Text-To-Speech (TTS) audio as a "helper" tool for training keyword spotters in these low-resource scenarios. In the experiments studied in this paper, the careful mixing of TTS audio with human speech audio during training led to a reduction of over 11% in the detection-error-tradeoff (DET) area under the curve (AUC) metric.

show abstract

Section: Model Architecture and Training Parametersmentioning

confidence: 99%

Exploring the application of synthetic audio in training keyword spotters

Werchniak

Chicote

Mishchenko

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several representative AC techniques like approximate arithmetic unit [5]- [11], voltage-over-scaling [12], [13], bit-width scaling (quantization) [13]- [17] can be either solely or hybridly applied for gaining efficiency. However, most of them [5], [11]- [14] focus on inference, whereas [8]- [10], [15]- [17] aim at training. Besides, the AC techniques are conventionally applied to forward propagation only, and back propagation rely on exact computation [5], [11]- [14] or additional training stages are required [5], [11].…”

Section: Introductionmentioning

confidence: 99%

“…However, most of them [5], [11]- [14] focus on inference, whereas [8]- [10], [15]- [17] aim at training. Besides, the AC techniques are conventionally applied to forward propagation only, and back propagation rely on exact computation [5], [11]- [14] or additional training stages are required [5], [11]. Although the inference and forward propagation enjoy AC benefits, the efficiency improvement of back propagation in training has less explored despite its importance.…”

Section: Introductionmentioning

confidence: 99%

“…Compared to common FP32, ImageNet and Pascal VOC enjoy 57%-63% energy reduction. [5], [11]- [14], and the back propagation still relies on FP32 to compensate the error induced in the forward propagation, or [5], [11] execute additional training stages. Besides, the main purpose of [5], [11]- [14] is to perform inference efficiently exploiting approximations for forward computation.…”

Section: Introductionmentioning

confidence: 99%

“…[5], [11]- [14], and the back propagation still relies on FP32 to compensate the error induced in the forward propagation, or [5], [11] execute additional training stages. Besides, the main purpose of [5], [11]- [14] is to perform inference efficiently exploiting approximations for forward computation. Efficiency improvement of back propagation in training is beyond their interests.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Minimizing Energy of DNN Training with Adaptive Bit-Width and Voltage Scaling

Cheng

Hashimoto

2021

2021 IEEE International Symposium on Circuits and Systems (ISCAS)

View full text Add to dashboard Cite

Training DNN mostly relies on GPUs with FP32 format. While FP16 is acknowledged for its advantage of high computation and memory efficiencies for training DNN, the training must be accompanied with techniques dedicated for a particular dataset. Therefore, a hardware engine with a configurable bit-width feature is desirable for covering any datasets and applications. This work proposes an adaptive bit-width and voltage scaling (ABVS) scheme for DNN training. The key idea is to increase fraction bit-width (FB) gradually from a small value according to current training quality (e.g., accuracy, mAP). Since less FB achieves shorter hardware latency, this training scheme concurrently adapts bit-width and voltage scaling and intensify energy reduction. Experimental results show that the ABVS scheme achieves the comparable quality to FP32 with at most 0.5% accuracy drop, but up to 63% energy reduction.Index Terms-deep learning training, floating-point, configurable bit-width, bit-width scaling, voltage scaling

show abstract