Wise teachers train better DNN acoustic models

Abstract

Automatic speech recognition is becoming more ubiquitous as recognition performance improves, capable devices increase in number, and areas of new application open up. Neural network acoustic models that can utilize speaker-adaptive features, have deep and wide layers, or more computationally expensive architectures, for example, often obtain best recognition accuracy but may not be suitable for the given budget of computational and storage resources or latency required by the deployed system. We explore a straightforward training approach which takes advantage of highly accurate but expensive-to-evaluate neural network acoustic models by using their outputs to relabel training examples for easier-to-deploy models. Experiments on a large vocabulary continuous speech recognition task offer relative reductions in word error rate of up to 16.7 % over training with the hard aligned labels by effectively making use of large amounts of additional untranscribed data. Somewhat remarkably, the approach works well even when only two output classes are present. Experiments on a voice activity detection task give relative reductions in equal error rate of up to 11.5 % when using a convolutional neural network to relabel training examples for a feedforward neural network. An investigation into the hidden layer weight matrices finds that soft target-trained networks tend to produce weight matrices having fuller rank and slower decay in singular values than their hard target-trained counterparts, suggesting that more of the network's capacity is utilized for learning additional information giving better accuracy.

Exploiting Eigenposteriors for Semi-Supervised Training of DNN Acoustic Models with Sequence Discrimination

“…Earlier studies investigate student-teacher framework to enable model compression and encapsulating the information of multiple models into a single network [12,15]. This approach was also found beneficial for semi-supervised training exploiting untranscribed training data [14], although the investigation is limited to the in-domain data matching the initial transcribed speech used for supervised training. Semi-supervised training [16,17,18,19] has been popular for low-resource tasks where cheap-to-obtain untranscribed data is readily available.…”

“…We exploit the framework of student-teacher DNN training that has been recognized promising for knowledge transfer and distillation [11,12,13,14]. The basic idea of the studentteacher DNN training is that a teacher DNN (often trained with hard targets) provides soft targets for training a student DNN.…”

Genetic Algorithm for Combined Speaker and Speech Recognition using Deep Neural Networks

“…We have used the trainlm training function because it is the fastest back propagation algorithm. It is based on the Levenberg-Marquardt optimization algorithm [31]- [34].…”

Far-Field ASR Using Low-Rank and Sparse Soft Targets from Parallel Data

“…Soft targets have been previously used for DNN knowledge distillation by model compression [4,5] and knowledge transfer [6,7]. In current work, they refer to the senone posteriors probabilities generated by an already trained DNN model on close-talk speech data.…”