Effective Transfer Learning with Label-Based Discriminative Feature Learning

Abstract

The performance of natural language processing with a transfer learning methodology has improved by applying pre-training language models to downstream tasks with a large number of general data. However, because the data used in pre-training are irrelevant to the downstream tasks, a problem occurs in that it learns general features rather than those features specific to the downstream tasks. In this paper, a novel learning method is proposed for embedding pre-trained models to learn specific features of such tasks. The proposed method learns the label features of downstream tasks through contrast learning using label embedding and sampled data pairs. To demonstrate the performance of the proposed method, we conducted experiments on sentence classification datasets and evaluated whether the features of the downstream tasks have been learned through a PCA and a clustering of the embeddings.

Deep transfer learning from limited source for abdominal CT and MR image segmentation

“…Unsupervised transfer learning involves transferring knowledge from a labeled source domain to an unlabeled target domain. Techniques such as domain adaptation or unsupervised pre-training are often utilized, allowing the model to extract relevant features from the source domain and apply them to the target domain [11]. Semi-supervised transfer learning combines labeled data from both the target and source domains.…”