Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models

Abstract

We present results from Alexa speech teams on semi-supervised learning (SSL) of acoustic models (AM) with experiments spanning over 3000 hours of GPU time, making our study one of the largest of its kind. We discuss SSL for AMs in a small footprint setting, showing that a smaller capacity model trained with 1 million hours of unsupervised data can outperform a baseline supervised system by 14.3% word error rate reduction (WERR). When increasing the supervised data to seven-fold, our gains diminish to 7.1% WERR; to improve SSL efficiency at larger supervised data regimes, we employ a step-wise distillation into a smaller model, obtaining a WERR of 14.4%. We then switch to SSL using larger student models in low data regimes; while learning efficiency with unsupervised data is higher, student models may outperform teacher models in such a setting. We develop a theoretical sketch to explain this behavior.

Noisy Training Improves E2E ASR for the Edge

“…Automatic Speech Recognition (ASR) enables fast and accurate transcriptions of voice commands and dictations on edge devices; examples of on-device ASR applications include dictation for Google keyboard [1,2], voice commands for Apple Siri [3], and Amazon Alexa [4], etc. Past work developed streaming End-to-End (E2E) all-neural ASR models that run compactly on edge devices [2,5,6,7].…”