We propose a novel adversarial multi-task learning scheme, aiming at actively curtailing the inter-talker feature variability while maximizing its senone discriminability so as to enhance the performance of a deep neural network (DNN) based ASR system. We call the scheme speaker-invariant training (SIT). In SIT, a DNN acoustic model and a speaker classifier network are jointly optimized to minimize the senone (tied triphone state) classification loss, and simultaneously mini-maximize the speaker classification loss. A speakerinvariant and senone-discriminative deep feature is learned through this adversarial multi-task learning. With SIT, a canonical DNN acoustic model with significantly reduced variance in its output probabilities is learned with no explicit speaker-independent (SI) transformations or speaker-specific representations used in training or testing. Evaluated on the CHiME-3 dataset, the SIT achieves 4.99% relative word error rate (WER) improvement over the conventional SI acoustic model. With additional unsupervised speaker adaptation, the speaker-adapted (SA) SIT model achieves 4.86% relative WER gain over the SA SI acoustic model.Index Termsspeaker-invariant training, adversarial learning, speech recognition, deep neural networks * Zhong Meng performed the work while he was a research intern at Microsoft AI and Research, Redmond, WA, USA.

Because of its streaming nature, recurrent neural network transducer (RNN-T) is a very promising end-to-end (E2E) model that may replace the popular hybrid model for automatic speech recognition. In this paper, we describe our recent development of RNN-T models with reduced GPU memory consumption during training, better initialization strategy, and advanced encoder modeling with future lookahead. When trained with Microsoft's 65 thousand hours of anonymized training data, the developed RNN-T model surpasses a very well trained hybrid model with both better recognition accuracy and lower latency. We further study how to customize RNN-T models to a new domain, which is important for deploying E2E models to practical scenarios. By comparing several methods leveraging text-only data in the new domain, we found that updating RNN-T's prediction and joint networks using text-to-speech generated from domain-specific text is the most effective.

Unsupervised domain adaptation of speech signal aims at adapting a well-trained source-domain acoustic model to the unlabeled data from target domain. This can be achieved by adversarial training of deep neural network (DNN) acoustic models to learn an intermediate deep representation that is both senone-discriminative and domain-invariant. Specifically, the DNN is trained to jointly optimize the primary task of senone classification and the secondary task of domain classification with adversarial objective functions. In this work, instead of only focusing on learning a domain-invariant feature (i.e. the shared component between domains), we also characterize the difference between the source and target domain distributions by explicitly modeling the private component of each domain through a private component extractor DNN. The private component is trained to be orthogonal with the shared component and thus implicitly increases the degree of domain-invariance of the shared component. A reconstructor DNN is used to reconstruct the original speech feature from the private and shared components as a regularization. This domain separation framework is applied to the unsupervised environment adaptation task and achieved 11.08% relative WER reduction from the gradient reversal layer training, a representative adversarial training method, for automatic speech recognition on CHiME-3 dataset.Index Termsrobust speech recognition, deep neural networks, domain adaptation, adversarial training, multi-task training * Zhong Meng performed the work while he was a research intern at Microsoft AI and Research, Redmond, WA.

Representing digital ink traces as points in a function space has proven useful for online recognition. Ink trace coordinates or their integral invariants are written as parametric functions and approximated by truncated orthogonal series. This representation captures the shape of the ink traces with a small number of coefficients in a form quite compact and independent of device resolution, and various geometric techniques may be employed for recognition. The simplicity and high performance of this method lead us to ask whether the same idea can be applied to another important aspect in online handwriting -the compression of digital ink strokes. We have investigated Chebyshev, Legendre and Legendre-Sobolev orthogonal polynomial bases as well as Fourier series and have found that Chebyshev representation is the most suitable apparatus for compressing digital curves. We obtain compression rates of 30× to 50× and have the added benefit that the LegendreSobolev form, used for recognition, may be obtained by a single linear transformation.

No abstract