ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization

Abstract

Recent advances on large-scale pre-training have shown great potentials of leveraging a large set of Pre-Trained Models (PTMs) for improving Out-of-Distribution (OoD) generalization, for which the goal is to perform well on possible unseen domains after fine-tuning on multiple training domains. However, maximally exploiting a zoo of PTMs is challenging since fine-tuning all possible combinations of PTMs is computationally prohibitive while accurate selection of PTMs requires tackling the possible data distribution shift for OoD tasks. In this work, we propose ZooD, a paradigm for PTMs ranking and ensemble with feature selection. Our proposed metric ranks PTMs by quantifying inter-class discriminability and inter-domain stability of the features extracted by the PTMs in a leave-one-domain-out cross-validation manner. The top-K ranked models are then aggregated for the target OoD task. To avoid accumulating noise induced by model ensemble, we propose an efficient variational EM algorithm to select informative features. We evaluate our paradigm on a diverse model zoo consisting of 35 models for various OoD tasks and demonstrate: (i) model ranking is better correlated with fine-tuning ranking than previous methods and up to 9859x faster than brute-force fine-tuning; (ii) OoD generalization after model ensemble with feature selection outperforms the state-of-the-art methods and the accuracy on most challenging task DomainNet is improved from 46.5% to 50.6%. Furthermore, we provide the fine-tuning results of 35 PTMs on 7 OoD datasets, hoping to help the research of model zoo and OoD generalization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/zood.

Decomposed adversarial domain generalization

“…PTMs for domain generalization. Methods leveraging pretraining models have shown promising improvements in domain generalization performance (Wiles et al, 2022;Arpit et al, 2021;Dong et al, 2022;Wortsman et al, 2022;Rame et al, 2022;Ramé et al, 2022). Among them, ensemble methods combined with PTMs show further advantages.…”

“…Arpit et al (2021) ensemble the predictions of moving average models. Recent methods Dong et al, 2022) further consider the ensemble of models with different architectures to exploit the growing large PTM hubs. Specifically, ensemble predictions of multiple different PTMs via instance-specific attention weights.…”

“…Such a zoo of PTMs thus enjoys both high transfer ability and diversity. By selecting optimal PTMs for given DG datasets from a zoo of PTMs, Dong et al (2022) boosted the state-of-the-art DG performance on some benchmarks for over 14%.…”

“…While utilizing PTMs has proven to be a promising approach for domain generalization, it remains unclear how to effectively leverage the diverse inductive biases present in different PTMs. Ensemble methods of PTMs have been explored (Dong et al, 2022;You et al, 2021), however, these methods typically only consider the top-performing models based on performance ranking scores. For exam-ple, Dong et al (2022) proposed a feature selection method on the concatenated features of the top-3 ranked PTMs.…”

“…Ensemble methods of PTMs have been explored (Dong et al, 2022;You et al, 2021), however, these methods typically only consider the top-performing models based on performance ranking scores. For exam-ple, Dong et al (2022) proposed a feature selection method on the concatenated features of the top-3 ranked PTMs. However, without incorporating diversity, such ensembles can perform worse than single models.…”