Patching open-vocabulary models by interpolating weights

Ilharco, Gabriel; Wortsman, Mitchell; Yitzhak, Gadre, Samir; Song, Shuran; Hajishirzi, Hannaneh; Kornblith, Simon; Farhadi, Ali; Schmidt, Ludwig

doi:10.48550/arxiv.2208.05592

Cited by 6 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This yields improved classification accuracy on IMAGENET. Along the same line, [24] fine-tune a model trained on IMAGENET on several new image classification datasets, and show that interpolating the original and fine-tuned parameters yields classifiers that perform well on all tasks.…”

Section: Related Workmentioning

confidence: 86%

“…In the following, we formally introduce the two main components of our procedure to merge adversarially robust models: (1) obtaining models which can interpolated by fine-tuning a single p -robust classifiers, and (2) interpolation of their weights to balance their different types of robustness. We highlight that our setup diverges from that of prior works about parameters averaging: in fact, both [24,46] combine models fine-tuned on the same task, i.e. achieving high classification accuracy of unperturbed images, either on a fixed dataset and different hyperparameter configurations [46], or varying datasets [24].…”

Section: Model Interpolation Across Different Tasksmentioning

confidence: 91%

“…We highlight that our setup diverges from that of prior works about parameters averaging: in fact, both [24,46] combine models fine-tuned on the same task, i.e. achieving high classification accuracy of unperturbed images, either on a fixed dataset and different hyperparameter configurations [46], or varying datasets [24]. In our case, the individual models are trained for robustness to different types of attacks, i.e.…”

Section: Model Interpolation Across Different Tasksmentioning

confidence: 94%

See 2 more Smart Citations

Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shifts

Croce¹,

Rebuffi²,

Shelhamer³

et al. 2023

Preprint

View full text Add to dashboard Cite

Adversarial training is widely used to make classifiers robust to a specific threat or adversary, such as p -norm bounded perturbations of a given p-norm. However, existing methods for training classifiers robust to multiple threats require knowledge of all attacks during training and remain vulnerable to unseen distribution shifts. In this work, we describe how to obtain adversarially-robust model soups (i.e., linear combinations of parameters) that smoothly trade-off robustness to different p -norm bounded adversaries. We demonstrate that such soups allow us to control the type and level of robustness, and can achieve robustness to all threats without jointly training on all of them. In some cases, the resulting model soups are more robust to a given p -norm adversary than the constituent model specialized against that same adversary. Finally, we show that adversarially-robust model soups can be a viable tool to adapt to distribution shifts from a few examples.

show abstract

Section: Related Workmentioning

confidence: 86%

Section: Model Interpolation Across Different Tasksmentioning

confidence: 91%

Section: Model Interpolation Across Different Tasksmentioning

confidence: 94%

See 1 more Smart Citation

Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shifts

Croce¹,

Rebuffi²,

Shelhamer³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Fine-Tuning Pre-Trained Representations. Several recent works study how to adapt pre-trained representations for downstream tasks (Kumar et al, 2022;Wortsman et al, 2022;Ilharco et al, 2022;Lee et al, 2022;Kirichenko et al, 2022;Goyal et al, 2022;Dong et al, 2022), motivated by the emergence of large pre-trained models (Radford et al, 2021;Brown et al, 2020) capable of zero-shot transfer. Most closely related to our work are a few concurrent works that find using the CLIP objective to fine-tune CLIP is more effective than alternative fine-tuning approaches (Goyal et al, 2022;Dong et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

Likelihood-Based Diverse Sampling for Trajectory Forecasting

Jason¹,

Inala

Jayaraman

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

We present Language-Image Value learning (LIV), a unified objective for vision-language representation and reward learning from action-free videos with text annotations. Exploiting a novel connection between dual reinforcement learning and mutual information contrastive learning, the LIV objective trains a multi-modal representation that implicitly encodes a universal value function for tasks specified as language or image goals. We use LIV to pre-train the first control-centric vision-language representation from large human video datasets such as EpicKitchen. Given only a language or image goal, the pre-trained LIV model can assign dense rewards to each frame in videos of unseen robots or humans attempting that task in unseen environments. Further, when some target domain-specific data is available, the same objective can be used to fine-tune and improve LIV and even other pre-trained representations for robotic control and reward specification in that domain. In our experiments on several simulated and real-world robot environments, LIV models consistently outperform the best prior input state representations for imitation learning, as well as reward specification methods for policy synthesis. Our results validate the advantages of joint vision-language representation and reward learning within the unified, compact LIV framework. Project website: penn-pal-lab.github.io/LIV

show abstract

“…However, we aim to study the fundamental question of how to finetune multi-modal (as opposed to uni-modal) models. A crucial difference between prior art and ours is the use of textual information, as all existing methods [41,100,111,113] repurpose additional text features as classifier weights instead of training samples. We demonstrate in this paper that crossmodal adaptation is not only more performant but can also benefit prior uni-modal approaches.…”

Section: Introductionmentioning

confidence: 99%

Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models

Zhang¹,

Yu²,

Kuang³

et al. 2023

Preprint

View full text Add to dashboard Cite

The ability to quickly learn a new task with minimal instruction -known as few-shot learning -is a central aspect of intelligent agents. Classical few-shot benchmarks make use of few-shot samples from a single modality, but such samples may not be sufficient to characterize an entire concept class. In contrast, humans use cross-modal information to learn new concepts efficiently. In this work, we demonstrate that one can indeed build a better visual dog classifier by reading about dogs and listening to them bark. To do so, we exploit the fact that recent multimodal foundation models such as CLIP are inherently cross-modal, mapping different modalities to the same representation space. Specifically, we propose a simple cross-modal adaptation approach that learns from few-shot examples spanning different modalities. By repurposing class names as additional one-shot training samples, we achieve SOTA results with an embarrassingly simple linear classifier for vision-language adaptation. Furthermore, we show that our approach can benefit existing methods such as prefix tuning, adapters, and classifier ensembling. Finally, to explore other modalities beyond vision and language, we construct the first (to our knowledge) audiovisual few-shot benchmark and use crossmodal training to improve the performance of both image and audio classification. Project site at link.

show abstract

Patching open-vocabulary models by interpolating weights

Cited by 6 publications

References 0 publications

Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shifts

Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shifts

Likelihood-Based Diverse Sampling for Trajectory Forecasting

Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models

Contact Info

Product

Resources

About