On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

Abstract

Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allowing a high degree of parameter sharing. Prior studies have shown that adapter-based tuning often achieves comparable results to finetuning. However, existing work only focuses on the parameter-efficient aspect of adapterbased tuning while lacking further investigation on its effectiveness. In this paper, we study the latter. We first show that adapterbased tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically compare the two tuning methods on several downstream NLP tasks and settings. We demonstrate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

“…The RoBERTa-large MNLI results of our adapter implementation is on par with the recent stateof-the-art Compacter adapters on T5 (Mahabadi et al, 2021), but generalization in both BERT and RoBERTa is overall worse than with vanilla finetuning. Following on the recent report of adapter efficacy in low-resource setting (He et al, 2021), we conducted an additional experiment with adapters and RoBERTa-large, where the model had to learn from a small, more informative subsample. At 1024 training examples adapters performed better when the MNLI subsample was diverse (selected with K-means-based clustering, see appendix D) rather than randomly selected: 80.7% vs 85%.…”

UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning

“…Adapter-tuning has shown to be on par with fine-tuning and sometimes exhibits better effectiveness in the low-resource setting (He et al, 2021). Later studies extend adapter-tuning to multi-lingual (Pfeiffer et al, 2021) andmulti-task (Karimi Mahabadi et al, 2021) settings, or further reduce the trainable parameters , which can be easily incorporated into UNIPELT as a replacement of the vanilla adapter-tuning.…”

“…We conduct extensive experiments on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al, 2019), which involves four types of natural language understanding tasks including linguistic acceptability (CoLA), sentiment analysis (SST-2), similarity and paraphrase tasks (MRPC, STS-B, QQP), and natural language inference (MNLI, QNLI, RTE). WNLI is omitted following prior studies (Houlsby et al, 2019;Devlin et al, 2019;He et al, 2021;Ben Zaken et al, 2021) due to its adversarial nature. Data Setup.…”

“…We sample a small subset of the training set for each task with size K = {100, 500, 1000}. As it is infeasible to submit a large number of runs to the GLUE leaderboard (2 submissions/day), we take 1,000 samples on the training set as the development set to select the best checkpoint and use the original development set as the test set following He et al (2021). Specifically, we randomly shuffle the training set with seed s, take the first K samples as the new training set, and the next 1,000 samples as the development set.…”

“…A more challenging yet barely studied problem is whether one can achieve better performance than fine-tuning with fewer parameters. Recent studies (He et al, 2021;Li and Liang, 2021; find that some PELT methods could be more effective than fine-tuning when the training data is limited, possibly due to the reduced risk of overfitting. However, as found in our analytical experiments, various PELT methods may exhibit diverse characteristics and perform rather differently on the same task, which makes it nontrivial to select the most appropriate method for a specific task, especially considering the fast-growing number of new PELT methods as well as downstream tasks.…”