An Information-Theoretic Approach to Transferability in Task Transfer Learning

Bao, Yajie; Li, Yang; Huang, Shao-Lun; Zhang, Lin; Zheng, Lizhong; Zamir, Amir; Guibas, Leonidas J.

doi:10.1109/icip.2019.8803726

Cited by 80 publications

(112 citation statements)

References 12 publications

Supporting

Mentioning

110

Contrasting

Order By: Relevance

“…Despite their general applicability in using unsupervised pre-trained models besides supervised ones and selecting the layer to transfer, their computational costs that are as high as fine-tuning with target labeled data exclude their applications to meet the urgent need of transferabilty estimation prior to fine-tuning. This work is more aligned with recent attempts towards computationally efficient transferability measures without training on target data [10][11][12][13][14]. The Earth Mover's Distance between features of the source and the target is used in [11].…”

Section: Related Workmentioning

confidence: 94%

“…To improve the performance on a target task and avoid negative transfer, there have been various works on transferability estimation between tasks [6,10,11,7,12,8,13,9], which we summarize in Table 1.…”

Section: Related Workmentioning

confidence: 99%

“…This research question drives the design of transferability estimation methods, including computationintensive [6][7][8][9] and computation-efficient ones [10][11][12][13][14]. The pioneering works [6,9] directly follow the definition of transfer learning to measure the transferability, and thereby require running fine-tuning on a target task with expensive parameter optimization.…”

Section: Introductionmentioning

confidence: 99%

“…The transferability is estimated as the negative conditional entropy between the labels of the two tasks in [13] and [12]. Bao et al [10] and You et al [14] solved two surrogate optimization problems to estimate the likelihood and marginalized likelihood of labeled target examples, under the assumption that a linear classifier is added on top of the pre-trained model. However, the computation efficiency comes at the price of failing to discriminate the transferability between layers, for both [12,13] that estimate with labels only and [10,14] that consider transferring the penultimate layer only.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Frustratingly Easy Transferability Estimation

Huang¹,

Wei²,

Rong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Transferability estimation has been an essential tool in selecting a pre-trained model and the layers of it to transfer, so as to maximize the performance on a target task and prevent negative transfer. Existing estimation algorithms either require intensive training on target tasks or have difficulties in evaluating the transferability between layers. We propose a simple, efficient, and effective transferability measure named TransRate. With single pass through the target data, TransRate measures the transferability as the mutual information between the features of target examples extracted by a pre-trained model and labels of them. We overcome the challenge of efficient mutual information estimation by resorting to coding rate that serves as an effective alternative to entropy. TransRate is theoretically analyzed to be closely related to the performance after transfer learning. Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 22 pre-trained models and 16 downstream tasks.

show abstract

Section: Related Workmentioning

confidence: 94%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Frustratingly Easy Transferability Estimation

Huang¹,

Wei²,

Rong³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, characterizing the task relatedness is a nontrivial problem since empirically evaluating the transfer performance of each task pair is computational expensive and time consuming. Recent analytical transferability metrics [9,1,5,8] are able to assess the transferability of a source model in a more efficient manner. In practice, we can not only apply transferability metrics to select the best source model, but also help ranking the highly transferable tasks for joint training [10] and multi-source feature fusion [8].…”

Section: Introductionmentioning

confidence: 99%

Transferability Estimation for Semantic Segmentation Task

Tan,

Li,

Huang

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Transferability estimation is a fundamental problem in transfer learning to predict how good the performance is when transferring a source model (or source task) to a target task. With the guidance of transferability score, we can efficiently select the highly transferable source models without performing the real transfer in practice. Recent analytical transferability metrics are mainly designed for image classification, and currently there is no specific investigation for the transferability estimation of semantic segmentation task, which is an essential problem in autonomous driving, medical image analysis, etc. Consequently, we further extend the recent analytical transferability metric OTCE (Optimal Transport based Conditional Entropy) score to the semantic segmentation task. The challenge in applying the OTCE score is the high dimensional segmentation output, which is difficult to find the optimal coupling between so many pixels under an acceptable computation cost. Thus we propose to randomly sample N pixels for computing OTCE score and take the expectation over K repetitions as the final transferability score. Experimental evaluation on Cityscapes, BDD100K and GTA5 datasets demonstrates that the OTCE score highly correlates with the transfer performance.

show abstract

A Transfer Machine Learning Matching Algorithm for Source and Target (TL-MAST)

Peters¹,

Hirt²

2020

Machine Learning, Optimization, and Data Science

View full text Add to dashboard Cite

An Information-Theoretic Approach to Transferability in Task Transfer Learning

Cited by 80 publications

References 12 publications

Frustratingly Easy Transferability Estimation

Frustratingly Easy Transferability Estimation

Transferability Estimation for Semantic Segmentation Task

A Transfer Machine Learning Matching Algorithm for Source and Target (TL-MAST)

Contact Info

Product

Resources

About