Transferability and Hardness of Supervised Classification Tasks

Abstract

We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Instead, we estimate these values using an information theoretic approach: treating training labels as random variables and exploring their statistics. When transferring from a source to a target task, we consider the conditional entropy between two such variables (i.e., label assignments of the two tasks). We show analytically and empirically that this value is related to the loss of the transferred model. We further show how to use this value to estimate task hardness. We test our claims extensively on three large scale data sets-CelebA (40 tasks), Animals with Attributes 2 (85 tasks), and Caltech-UCSD Birds 200 (312 tasks)-together representing 437 classification tasks. We provide results showing that our hardness and transferability estimates are strongly correlated with empirical hardness and transferability. As a case study, we transfer a learned face recognition model to CelebA attribute classification tasks, showing state of the art accuracy for tasks estimated to be highly transferable.

Probing transfer learning with a model of synthetic correlated datasets

“…On a different note, there have been few proposals of viable metrics for evaluating similarity between real datasets, with the goal of obtaining predictors of the associated transfer learning performance. Some of these architecture agnostic metrics evaluate dataset distances based on information theory, information geometry and optimal transport [53][54][55][56][57]. An interesting direction for future work would be to connect these distance metrics with the parametric transformations in the CHMM.…”

LogME: Practical Assessment of Pre-trained Models for Transfer Learning

“…The log expectation of the empirical predictor (LEEP) is used as a transferability measure. The LEEP method is closely related to Negative Conditional Entropy (NCE) proposed by Tran et al (2019), an informationtheoretic quantity (Cover, 1999) to study the transferability and hardness between classification tasks.…”

“…Despite its practical significance, there is limited guidance on task adaptive pre-trained model selection. Based on NCE (Tran et al, 2019), Nguyen et al (2020) recently studied the problem when both the pre-train task and the downstream task are classification. They construct an empirical predictor by estimating the joint distribution over the pretrained and target label spaces and take the performance of the empirical predictor (LEEP) to assess pre-trained models.…”

Exploiting a Zoo of Checkpoints for Unseen Tasks

“…Recently, NCE [21] and LEEP [8] measure task transferability by estimating an approximation of conditional entropy, H(y j |y i ) = H(y j ) − I(y i ; y j ). Ignoring H(y j ) and focusing on the mutual information term, we can understand these two methods as measuring the dependency between two target spaces Y i and Y j .…”