Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Abstract

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs-we call the results "model soups." When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pre-trained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. As a highlight, the resulting ViT-G model attains 90.94% top-1 accuracy on ImageNet, a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logit-ensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically.

Linear Connectivity Reveals Generalization Strategies

“…In some domain shift scenarios [25,46,15,14], OOD generalization behavior may therefore form basins. In current work, however, barriers are not observed in vision during transfer [35,52]. Is that because all vision models achieve a basin with good structural generalization-or none do?…”

“…The split between generalization strategies can potentially explain results from the bimodality of CoLA models [33] to wide variance on NLI diagnostic sets [31]. Because weight averaging can find parameter settings that fall on a barrier, we may even explain why weight averaging, which tends to perform well on vision tasks, fails in text classifiers [52]. Future work that distinguishes generalization strategy basins could improve the performance of such weight ensembling methods.…”

AdaMix: Mixture-of-Adapter for Parameter-efficient Tuning of Large Language Models

“…Prior work (Wortsman et al, 2022) show that averaging the weights of multiple models fine-tuned with different hyper-parameter configurations improves model performance. They analytically show the similarity in loss between weight-averaging (L AM W in our setting) and logit-ensembling (L Ens W in our setting) as a function of the flatness of the loss and confidence of the predictions.…”

“…MPQA (Wiebe et al, 2005) and Subj (Pang & Lee, 2004) are used for polarity and subjectivity detection, where we follow Matena and Raffel Matena & Raffel (2021) propose to merge pre-trained language models which are fine-tuned on various text classification tasks. Wortsman et al (2022) explores averaging model weights from various independent runs on the same task with different hyper-parameter configurations. Different from existing works, we focus on averaging weights of newly-added parameters for parameter-efficient fine-tuning purpose.…”

“…To address this, we develop a merging mechanism to average weights from differently learned adapter components to a single adapter that preserves the performance gains, while keeping the number of parameters and FLOPs also the same as that of a single adapter design. Our adapter merging is inspired by recent works on model weight averaging like model soups (Wortsman et al, 2022) and multi BERTs (Devlin et al, 2019). Such weight averaging of models with different random initialization has been shown to improve model performance in recent works (Matena & Raffel, 2021;Neyshabur et al, 2020;Frankle et al, 2020) that show the optimized models to lie in the same basin of error landscape.…”