Gpu-Accelerated Viterbi Exact Lattice Decoder for Batched Online and Offline Speech Recognition

Abstract

We present an optimized weighted finite-state transducer (WFST) decoder capable of online streaming and offline batch processing of audio using Graphics Processing Units (GPUs). The decoder is efficient in memory utilization, input/output bandwidth, and uses a novel Viterbi implementation designed to maximize parallelism. Memory savings enable the decoder to process larger graphs than previously possible while simultaneously supporting larger numbers of consecutive streams. GPU preprocessing of lattice segments enable intermediate lattice results to be returned to the requestor during streaming inference. Collectively, the proposed improvements achieve up to a 240x speedup over single core CPU decoding, and up to 40x faster decoding than the current state-of-the-art GPU decoder, while returning equivalent results. This architecture also makes deployment of production-grade models on hardware ranging from large data center servers to low-power edge devices practical.

Fast offline transformer‐based end‐to‐end automatic speech recognition for real‐world applications

“…To increase the throughput of RNN-based speech recognizers, Amodei et al, Braun et al, and Oh et al [33][34][35] used several batch processing approaches. In particular, Braun et al and Oh et al [34,35] aimed to accelerate GPU parallelization.…”

“…To increase the throughput of RNN-based speech recognizers, Amodei et al, Braun et al, and Oh et al [33][34][35] used several batch processing approaches. In particular, Braun et al and Oh et al [34,35] aimed to accelerate GPU parallelization. Seki et al [36,37] proposed a multiple-utterance multiple-hypothesis vectorized beam search in CTC-attention-based end-to-end speech recognition using a VGG-RNN-based encoder-decoder and showed the decoding throughput increased using a GPU.…”

The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage

“…• Transcribing the audio file with a pre-trained deep neural network acoustic model and n-gram language model via Kaldi's GPU-based decoder [16]. The output "hypothesis" transcript contains word-level timestamps.…”

“…Prior work has shown that once the acoustic model is accelerated on a GPU, roughly 90% of the run time will be spent in external language model decoding on the CPU [16]. Therefore, we were concerned that simply accelerating the acoustic model on a GPU would not give us meaningful overall speed up.…”

“…These challenges motivated us to use GPU-based external language model decoder in Kaldi [16], for which the GPU runs both the acoustic model inference and language model decoding, without having to save acoustic model logits to disk to be loaded later by a CPU-based language model decoder. We used only 4 NVIDIA T4 GPUs to align all of our data, with each running at 250x real-time-factor (i.e., 250 hours of audio could be transcribed in 1 hour of wall-clock time by one GPU).…”