Hybrid Transformer/CTC Networks for Hardware Efficient Voice Triggering

Abstract

We consider the design of two-pass voice trigger detection systems. We focus on the networks in the second pass that are used to re-score candidate segments obtained from the firstpass. Our baseline is an acoustic model(AM), with BiLSTM layers, trained by minimizing the CTC loss. We replace the BiLSTM layers with self-attention layers. Results on internal evaluation sets show that self-attention networks yield better accuracy while requiring fewer parameters. We add an autoregressive decoder network on top of the self-attention layers and jointly minimize the CTC loss on the encoder and the crossentropy loss on the decoder. This design yields further improvements over the baseline. We retrain all the models above in a multi-task learning(MTL) setting, where one branch of a shared network is trained as an AM, while the second branch classifies the whole sequence to be true-trigger or not. Results demonstrate that networks with self-attention layers yield ∼60% relative reduction in false reject rates for a given false-alarm rate, while requiring 10% fewer parameters. When trained in the MTL setup, self-attention networks yield further accuracy improvements. On-device measurements show that we observe 70% relative reduction in inference time. Additionally, the proposed network architectures are ∼5X faster to train.

Progressive Voice Trigger Detection: Accuracy vs Latency

“…p(VoiceTriggerPhoneSeq | audio). Although useful as demonstrated in previous work [14,15], the phonetic branch cannot be used to score the payload, where we are not sure what the phonetic content in the audio will be. The discriminative branch on other hand is trained to perform sequence classification.…”

“…The first-stage comprises a low-power detector that processes streaming audio and is always-on [12,13]. If a detection is made at the first stage, the detector marks the start and end points of the purported keyword segment (Figure 1) and the segment is then re-scored by larger, more complex models [14,15]. Note that this paper is concerned only with the larger models in the second-pass.…”

“…Following previous work [14,15,16], the input to the model is a 40-dimensional Mel-filterbank which is computed at a rate of 100 frames-per-second (FPS) from the input audio. We stack 7 contiguous frames together to form an input window and down-sample the sequence of windows by a factor of 3.…”

“…We use exactly the same phonetic training data as described in [14,15]. This dataset comprises 8000 hours of transcribed audio obtained from intended invocations of the voice assistant.…”

“…The positive set contains 140,000 examples while the negative set contains 40,000 utterances. In previous experiments [14,15], we only used audio segments that correspond to the trigger phrase. However in this study, we are also interested in making use of the audio following the trigger phrase.…”