Adaptation of Artificial Neural Networks Avoiding Catastrophic Forgetting

Abstract

In connectionist learning, one relevant problem is "catastrophic forgetting" that may occur when a network, trained with a large set of patterns, has to learn new input patterns, or has to be adapted to a different environment. The risk of catastrophic forgetting is particularly high when a network is adapted with new data that do not adequately represent the knowledge included in the original training data.Two original solutions are proposed to reduce the risk that the network focuses on new data only, loosing its generalization capability. The first one, Conservative Training, is a variant to the target assignment policy, while the second approach, Support Vector Rehearsal, selects from the training set the patterns that lay near the borders of the classes not included in the adaptation set. These patterns are used as sentinels that try to keep unchanged the original boundaries of these classes. Moreover, we investigated the extension of the classical approach consisting in applying linear transformations not only to the input features, but also to the outputs of the internal layers. The motivation is that the outputs of an internal layer represent a projection of the input pattern into a space where it should be easier to learn the classification or transformation expected at the output of the network.We illustrate the problems using an artificial test-bed, and apply our techniques to a set of adaptation tasks in the domain of Automatic Speech Recognition (ASR) based on Artificial Neural Networks. Supervised ASR adaptation experiments with several corpora and for different adaptation types are described. We report on the adaptation potential of different techniques, and on the generalization capability of the adapted networks. The results show that the combination of the proposed approaches mitigates the catastrophic forgetting effects, and always outperforms the use of the classical transformations in the feature space.

KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition

“…Another popular category of adaptation techniques is conservative training (CT) [22]. CT can be achieved by adding regularizations to the adaptation criterion.…”

Fast speaker adaptation using extended diagonal linear transformation for deep neural networks

“…Feature‐based adaptation methods extract features by minimizing an individual speaker's characteristics using feature transformation methods , whereas a model‐adaptation is a method for transforming a speaker‐independent (SI) model into a more speaker‐dependent (SD) version. Model‐adaptation methods include regularization methods that decrease the overfitting by preventing many deviations from the SI model; learning speaker information such as a speaker code , bases , and i‐vectors that can effectively combine with general features and other adaptation methods; and transformation‐based techniques that convert the DNN layer into a speaker adapted layer. In particular, whereas feature adaptation affects all senones, by applying relatively small transformation parameters to the input layer, a model adaptation can be applied to all other layers.…”

Exploring GMM-derived Features for Unsupervised Adaptation of Deep Neural Network Acoustic Models

“…Linear transformation can be applied at different levels of the DNN system: to the input features, as in linear input network transformation (LIN) [6] or featurespace discriminative linear regression (fDLR); to the activations of hidden layers, as in linear hidden network transformation (LHN) [6]; or to the softmax layer, as in LON or in output-feature discriminative linear regression. The second type of adaptation consists in re-training the entire network or only a part of it using special regularization techniques for improving generalization, such as L2prior regularization [7], Kullback-Leibler divergence regularization [8], conservative training [9]. The concept of multi-task learning (MTL) has recently been applied to the task of speaker adaptation and has been shown to improve the performance of different model-based DNN adaptation techniques, such as LHN and learning speaker-specific hidden unit contributions [10].…”