Transferability Estimation using Bhattacharyya Class Separability

Abstract

Transfer learning has become a popular method for leveraging pre-trained models in computer vision. However, without performing computationally expensive finetuning, it is difficult to quantify which pre-trained source models are suitable for a specific target task, or, conversely, to which tasks a pre-trained source model can be easily adapted to. In this work, we propose Gaussian Bhattacharyya Coefficient (GBC), a novel method for quantifying transferability between a source model and a target dataset. In a first step we embed all target images in the feature space defined by the source model, and represent them with per-class Gaussians. Then, we estimate their pairwise class separability using the Bhattacharyya coefficient, yielding a simple and effective measure of how well the source model transfers to the target task. We evaluate GBC on image classification tasks in the context of dataset and architecture selection. Further, we also perform experiments on the more complex semantic segmentation transferability estimation task. We demonstrate that GBC outperforms state-of-the-art transferability metrics on most evaluation criteria in the semantic segmentation settings, matches the performance of top methods for dataset transferability in image classification, and performs best on architecture selection problems for image classification.

How stable are Transferability Metrics evaluations?

“…We vary each of these components in a largescale systematic study of two scenarios: selecting good source datasets for semantic segmentation, and selecting good source model architectures for image classification. In total we construct a large set of 715k experiments, several orders of magnitude larger than previous works [56,47,86,58,6]. Based on these experiments: (A) we demonstrate that even small variations to an experimental setup leads to very different conclusions about the superiority of a transferability metric over another.…”

“…Transferability metrics are then computed using the embeddings and their corresponding labels. These methods include GBC [58], LogME [86], and N LEEP [48]. To conclude, optimal transport-based methods [72,4] develop cost functions usable within the optimal transport framework to determine transferability between two datasets.…”

“…Transferability metrics [6,47,56,72,74,86,58,2] provide heuristics for selecting the most suitable source models for a given target dataset, without explicitly fine-tuning them all. These methods generally work by applying a source model to the target dataset to compute embeddings or predictions.…”

“…Transferability metrics is a maturing field with numerous contributions and increasing interest [6,47,56,72,74,86,58,2]. However, existing works rely on custom experimental settings without standardized benchmarks, which leads to inconsistent conclusions across different papers.…”

“…For example, while NCE [74] consistently outperforms LEEP [56] in [86], LEEP outperforms NCE in both [72] and [56]. Furthermore LogME outperforms LEEP [86], but LEEP outperforms LogME in [58]. This raises questions about how stable conclusions are across different experimental setups and if there truly is a single best transferability metric.…”