Bifocal Neural ASR: Exploiting Keyword Spotting for Inference Optimization

Abstract

We present Bifocal RNN-T, a new variant of the Recurrent Neural Network Transducer (RNN-T) architecture designed for improved inference time latency on speech recognition tasks. The architecture enables a dynamic pivot for its runtime compute pathway, namely taking advantage of keyword spotting to select which component of the network to execute for a given audio frame. To accomplish this, we leverage a recurrent cell we call the Bifocal LSTM (BF-LSTM), which we detail in the paper. The architecture is compatible with other optimization strategies such as quantization, sparsification, and applying time-reduction layers, making it especially applicable for deployed, real-time speech recognition settings. We present the architecture and report comparative experimental results on voice-assistant speech recognition tasks. Specifically, we show our proposed Bifocal RNN-T can improve inference cost by 29.1% with matching word error rates and only a minor increase in memory size.

Compute Cost Amortized Transformer for Streaming ASR

“…which includes the standard neural transducer loss [21] and an added compute cost penalty term Lcompute. Accounted for by the cumulative number of FLOPs across the components of the network for a streaming sequence, Lcompute drives more computation cost reduction while maintaining predictive performance of the model.…”

“…To address these motivations, in this work, we present a Transformer architecture which is able to amortize compute cost on demand at inference by dynamically activating compute components conditioned on input frames. Our work is inspired by prior investigations of dynamic compute for speech processing which have yielded notable results [21,22,23,24]. Considering speech examples can be lengthy streaming sequences with many frames of different levels of inherent complexity (e.g.…”

“…consider silent frames between acoustically rich segments), dynamic computing enables a desired balance between accuracy and efficiency by altering the compute expenditure each frame. Unlike existing methods [21,23,24] which bifurcate ASR into only two fixed branched encoder networks, with our approach, an exponential family of dynamic branches are generated from a single Transformer encoder which significantly expands the modeling space and boosts the adaptability and expressivity of the model. Lightweight prediction arbitrators (e.g.…”

Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

“…We build a conventional RNN-T consisting of 5 LSTM encoding layers and 2 LSTM decoding layers with 1024 hidden units per layer and a fully connected joint layer. Furthermore, we benchmark GQ on an RNN-T variant with a branched encoder, named Bifocal RNN-T [18]. It has 2 encoders of different computational complexity and decides on-the-fly which encoder to use per input frame.…”

“…Under GQ, quantization centroids are self-adjustable but in a µ-Law constrained space. As a proof-of-concept, we adopt the ASR task and conduct experiments on both the LibriSpeech and de-identified far-field datasets to evaluate GQ on three major end-to-end ASR architectures, namely conventional Recurrent Neural Network Transducer (RNN-T) [17], Bifocal RNN-T [18], and Conformer [19] [20]. Our results show that in all three architectures, GQ yields little to no accuracy loss when compressing models to S8B or even sub-5-bit (5-bit or lower).…”