Unsupervised adaptation with domain separation networks for robust speech recognition

Abstract

Unsupervised domain adaptation of speech signal aims at adapting a well-trained source-domain acoustic model to the unlabeled data from target domain. This can be achieved by adversarial training of deep neural network (DNN) acoustic models to learn an intermediate deep representation that is both senone-discriminative and domain-invariant. Specifically, the DNN is trained to jointly optimize the primary task of senone classification and the secondary task of domain classification with adversarial objective functions. In this work, instead of only focusing on learning a domain-invariant feature (i.e. the shared component between domains), we also characterize the difference between the source and target domain distributions by explicitly modeling the private component of each domain through a private component extractor DNN. The private component is trained to be orthogonal with the shared component and thus implicitly increases the degree of domain-invariance of the shared component. A reconstructor DNN is used to reconstruct the original speech feature from the private and shared components as a regularization. This domain separation framework is applied to the unsupervised environment adaptation task and achieved 11.08% relative WER reduction from the gradient reversal layer training, a representative adversarial training method, for automatic speech recognition on CHiME-3 dataset.Index Termsrobust speech recognition, deep neural networks, domain adaptation, adversarial training, multi-task training * Zhong Meng performed the work while he was a research intern at Microsoft AI and Research, Redmond, WA.

Conditional Teacher-student Learning

“…ASR suffers from performance degradation when a well-trained acoustic model is applied in a new domain [19]. T/S learning [3,8,9] and adversarial learning [20,21,22,23,24] are two effective approaches that can suppress this domain mismatch by adapting a source-domain acoustic model to target-domain speech. T/S learning is more suited for the situation where unlabeled parallel data is available for adaptation, 2 in which a sequence of source-domain speech features is fed as the input to a source-domain teacher model and a parallel sequence of target-domain features is at the input to the target-domain student model to optimize the student model parameters by minimizing the T/S loss in Eq.…”

Cycle-Consistent Speech Enhancement

“…To improve the ASR performance, we enhance the 9137 noisy utterances in CHiME-3 with ACSE and re-train the clean DNN-HMM acoustic model in [31]. We use the same senone-level forced alignments as the clean model for re-training.…”

Domain Robust Feature Extraction for Rapid Low Resource ASR Development

“…The authors in [15,16] adapt their acoustic model to test data by learning an adaptation function between the hidden unit contributions of the training data and the development data. However, as pointed by [17], these methods require reliable tri-phone alignment and this may not always be successful in a mismatched condition.…”

“…Recently, there has been a lot of work [17,18,19] using adversarial training to adapt to the target domain data in a completely unsupervised way. The authors in [18] and [19] use a gradient reversal layer to train a domain classifier and try to learn domain invariant representations by passing a negative gradient on classification of domains.…”