Domain Expansion in DNN-Based Acoustic Models for Robust Speech Recognition

Abstract

Training acoustic models with sequentially incoming datawhile both leveraging new data and avoiding the forgetting effect -is an essential obstacle to achieving human intelligence level in speech recognition. An obvious approach to leverage data from a new domain (e.g., new accented speech) is to first generate a comprehensive dataset of all domains, by combining all available data, and then use this dataset to retrain the acoustic models. However, as the amount of training data grows, storing and retraining on such a large-scale dataset becomes practically impossible. To deal with this problem, in this study, we study several domain expansion techniques which exploit only the data of the new domain to build a stronger model for all domains. These techniques are aimed at learning the new domain with a minimal forgetting effect (i.e., they maintain original model performance). These techniques modify the adaptation procedure by imposing new constraints including (1) weight constraint adaptation (WCA): keeping the model parameters close to the original model parameters; (2) elastic weight consolidation (EWC): slowing down training for parameters that are important for previously established domains; (3) soft KL-divergence (SKLD): restricting the KL-divergence between the original and the adapted model output distributions; and (4) hybrid SKLD-EWC: incorporating both SKLD and EWC constraints. We evaluate these techniques in an accent adaptation task in which we adapt a deep neural network (DNN) acoustic model trained with native English to three different English accents: Australian, Hispanic, and Indian. The experimental results show that SKLD significantly outperforms EWC, and EWC works better than WCA. The hybrid SKLD-EWC technique results in the best overall performance.

Accented Speech Recognition: A Survey

“…This can be done through KL-divergence [9] between the adapted model and the baseline model, elastic weight consolidation (EWC), and weight constraint adaptation (WCA). Ghorbani et al [16] inves-tigates several of these methods for one transfer step finding a hybrid of soft-KL-divergence and EWC to be the best, while Houston [17] looks at the performance over multiple transfer steps finding that EWC covers 65% of the gap between a finetuned and a pooled (best-case) model across all accents. An advantage of these regularization techniques is that the original data need not be present in order to adapt and improve the model.…”

Analyzing Large Receptive Field Convolutional Networks for Distant Speech Recognition

“…Convolutional neural networks are one of many other machine learning strategies adapted for acoustic modeling to handle the long-term dependencies in ASR. Similar to RNNs, CNNs have also shown significant improvements in ASR performance over FC-DNNs [13,14,15,16]. Recent research also shows that the use of residual connections can train deeper CNN architectures in a more efficient way compared to RNNs [17].…”

Updating Only Encoders Prevents Catastrophic Forgetting of End-to-End ASR Models

“…Architecture-based methods [6,8] dynamically expand a model architecture for new data. In the field of ASR, various studies [9,10,11] have demonstrated the benefits of the incremental learning approach. Fu et al [12] proposed an incremental learning algorithm for end-to-end ASR that uses attention distillation and knowledge distillation [13] to prevent catastrophic forgetting.…”

“…Fu et al [12] proposed an incremental learning algorithm for end-to-end ASR that uses attention distillation and knowledge distillation [13] to prevent catastrophic forgetting. Such meth-ods require retaining the previous model [6,12] or adding extra parameters of the same size as the model for optimization [8,9]. Therefore, adapting recent end-to-end ASR models with a huge amount of parameters is computationally expensive.…”