Towards Making Deep Transfer Learning Never Hurt

Abstract

Transfer learning have been frequently used to improve deep neural network training through incorporating weights of pre-trained networks as the starting-point of optimization for regularization. While deep transfer learning can usually boost the performance with better accuracy and faster convergence, transferring weights from inappropriate networks hurts training procedure and may lead to even lower accuracy. In this paper, we consider deep transfer learning as minimizing a linear combination of empirical loss and regularizer based on pre-trained weights, where the regularizer would restrict the training procedure from lowering the empirical loss, with conflicted descent directions (e.g., derivatives).Following the view, we propose a novel strategy making regularization-based Deep Transfer learning Never Hurt (DTNH) that, for each iteration of training procedure, computes the derivatives of the two terms separately, then re-estimates a new descent direction that does not hurt the empirical loss minimization while preserving the regularization affects from the pre-trained weights. Extensive experiments have been done using common transfer learning regularizers, such as L 2 -SP and knowledge distillation, on top of a wide range of deep transfer learning benchmarks including Caltech, MIT indoor 67, CIFAR-10 and ImageNet. The empirical results show that the proposed descent direction estimation strategy DTNH can always improve the performance of deep transfer learning tasks based on all above regularizers, even when transferring pre-trained weights from inappropriate networks. All in all, DTNH strategy can improve state-of-the-art regularizers in all cases with 0.1%-7% higher accuracy in all experiments.

Long-Term Ship Position Prediction Using Automatic Identification System (AIS) Data and End-to-End Deep Learning

“…However, all the models were trained with eight years of data and are thus still capable of producing accurate predictions. The dataset size affects deep learning performance, as a small amount of training data may degrade the performance [51]. We recreated the DL model for the Malacca Strait (the area with the most extensive dataset size) for each variation in the training set to show the effect of the data size on the model performance (see Figure 11).…”

Non-binary deep transfer learning for image classification

“…4. L2SP regularization, where the weights are decayed towards their pre-trained values rather than 0 during fine-tuning, improves performance when the source and target dataset are closely related, but hinders it when they are less related [21,20,37] 5. Momentum should be lower for more closely related source and target datasets [18].…”

“…They showed that a high level of regularization decaying towards the pre-trained weights is beneficial on these datasets. It has since been shown that the L2SP regularizer can result in minimal improvement or even worse performance when the source and target datasets are less related [18,37].…”

“…However, there are many real world scenarios where the large amounts of training data required to obtain the best performance cannot be met or are prohibitively expensive. Transfer learning has been shown to improve performance in a wide variety of computer vision tasks, particularly when the source and target tasks are closely related and the target task is small [28,22,6,8,33,27,21,20,37]. It has become standard practice to pre-train on Imagenet 1K for many different tasks where the available labeled datasets are orders of magnitude smaller than Imagenet 1K [21,20,37,24,19,27,25,26,9].…”

“…Transfer learning has been shown to improve performance in a wide variety of computer vision tasks, particularly when the source and target tasks are closely related and the target task is small [28,22,6,8,33,27,21,20,37]. It has become standard practice to pre-train on Imagenet 1K for many different tasks where the available labeled datasets are orders of magnitude smaller than Imagenet 1K [21,20,37,24,19,27,25,26,9]. Several papers published in recent years have questioned this established paradigm.…”