OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations

Yang, Tan; Li, Yang; Huang, Shao-Lun

doi:10.1109/cvpr46437.2021.01552

Cited by 41 publications

(57 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent analytical transferability metrics [3,21,29,30] mostly avoid the expensive computation for retraining the source model and can efficiently estimate the transferability, which is useful in source model selection. However, they still have limitations.…”

Section: Related Workmentioning

confidence: 99%

“…Transferability characterizes such relationship via quantitatively evaluating how easy it is to transfer the knowledge learned from a source task to the target task. In practical scenarios [3,21,29,30], we can apply a transferability metric to directly select the best source Figure 1: Transfer 40 source models (randomly generated 50-categories classification tasks, corresponding to each point in the figure) from Clipart, Painting, Quickdraw, Sketch domains to a target task (25-categories) in Real domain, which demonstrates that it is unreliable to perform source model selection according to the source model accuracy, but our JC-NCE score can predict the transfer performance more accurately. model for a target task rather than trying each source model on the target data, which involves expensive computation.…”

Section: Introductionmentioning

confidence: 99%

“…model for a target task rather than trying each source model on the target data, which involves expensive computation. In addition, transferability can help prioritize different tasks for joint training [34] and multi-source feature fusion [29].…”

Section: Introductionmentioning

confidence: 99%

“…Meanwhile, previous empirical transferability metrics [1,33,34] suffer heavy computation burdens in retraining the source model to obtain the training loss or validation accuracy for indicating transferability. Recent analytical transferability metrics [3,21,29,30] are evidently more efficient to compute from practical data, but there also exists some drawbacks, e.g., strict data assumptions [3,30], insufficient performance [21]. And the state-of-the-art method OTCE [29] requires auxiliary tasks with known transfer accuracy for calculating the coefficients of a linear model, which involves extra computations and restricts its application scenarios.…”

Section: Introductionmentioning

confidence: 99%

“…Consequently, Tan et al [29] also propose a simplified version of OTCE, namely OTbased NCE score, that does not depend on auxiliary tasks. It builds a soft correspondence between source and target data via solving an Optimal Transport (OT) [14] problem, and then use the Negative Conditional Entropy (NCE) between the coupled source and target labels to characterize transferability.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Practical Transferability Estimation for Image Classification Tasks

Tan,

Li,

Huang

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Transferability estimation is an essential problem in transfer learning to predict how good the performance is when transferring a source model (or source task) to a target task. Recent analytical transferability metrics have been widely used for source model selection and multi-task learning. A major challenge is how to make transfereability estimation robust under the cross-domain cross-task settings. The recently proposed OTCE score solves this problem by considering both domain and task differences, with the help of transfer experiences on auxiliary tasks, which causes an efficiency overhead. In this work, we propose a practical transferability metric called JC-NCE score that dramatically improves the robustness of the task difference estimation in OTCE, thus removing the need for auxiliary tasks. Specifically, we build the joint correspondences between source and target data via solving an optimal transport problem with a ground cost considering both the sample distance and label distance, and then compute the transferability score as the negative conditional entropy of the matched labels. Extensive validations under the intra-dataset and inter-dataset transfer settings demonstrate that our JC-NCE score outperforms the auxiliary-task free version of OTCE for 7% and 12%, respectively, and is also more robust than other existing transferability metrics on average.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Practical Transferability Estimation for Image Classification Tasks

Tan,

Li,

Huang

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

A Multimodal Protein Representation Framework for Quantifying Transferability Across Biochemical Downstream Tasks

Zhang

et al. 2023

Advanced Science

View full text Add to dashboard Cite

Proteins are the building blocks of life, carrying out fundamental functions in biology. In computational biology, an effective protein representation facilitates many important biological quantifications. Most existing protein representation methods are derived from self-supervised language models designed for text analysis. Proteins, however, are more than linear sequences of amino acids. Here, a multimodal deep learning framework for incorporating ≈1 million protein sequence, structure, and functional annotation (MASSA) is proposed. A multitask learning process with five specific pretraining objectives is presented to extract a fine-grained protein-domain feature. Through pretraining, multimodal protein representation achieves state-of-the-art performance in specific downstream tasks such as protein properties (stability and fluorescence), protein-protein interactions (shs27k/shs148k/string/skempi), and protein-ligand interactions (kinase, DUD-E), while achieving competitive results in secondary structure and remote homology tasks. Moreover, a novel optimal-transport-based metric with rich geometry awareness is introduced to quantify the dynamic transferability from the pretrained representation to the related downstream tasks, which provides a panoramic view of the step-by-step learning process. The pairwise distances between these downstream tasks are also calculated, and a strong correlation between the inter-task feature space distributions and adaptability is observed.

show abstract