Revisiting Pretraining with Adapters

Kim, Seungwon; Shum, Alex; Susanj, Nathan; Hilgart, Jonathan

doi:10.18653/v1/2021.repl4nlp-1.11

Cited by 7 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TAPT could alternatively be performed by freezing the base model and training the adapters with an MLM objective. Despite being a faster approach, this has been found to sometimes decrease performance (Kim et al, 2021). Adapters Our multilingual system used the Pfeiffer bottleneck adapter configuration, with a reduction factor of 8, which for XLM-R LARGE corresponds to a bottleneck hidden size of 128.…”

Section: Multilingual Systemmentioning

confidence: 99%

SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification

Wu¹,

Razuvayevskaya²,

Freddy³

et al. 2023

Proceedings of the the 17th International Workshop on Semantic Evaluation (SemEval-2023)

View full text Add to dashboard Cite

This paper describes our approach for SemEval-2023 Task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. For Subtask 1 (News Genre), we propose an ensemble of fully trained and adapter mBERT models which was ranked joint-first for German, and had the highest mean rank of multi-language teams. For Subtask 2 (Framing), we achieved first place in 3 languages, and the best average rank across all the languages, by using two separate ensembles: a monolingual RoBERTa-MUPPET LARGE and an ensemble of XLM-RoBERTa LARGE with adapters and task adaptive pretraining. For Subtask 3 (Persuasion Techniques), we trained a monolingual RoBERTa-Base model for English and a multilingual mBERT model for the remaining languages, which achieved top 10 for all languages, including 2nd for English. For each subtask, we compared monolingual and multilingual approaches, and considered class imbalance techniques. 1

show abstract

Section: Multilingual Systemmentioning

confidence: 99%

SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification

Wu¹,

Razuvayevskaya²,

Freddy³

et al. 2023

Proceedings of the the 17th International Workshop on Semantic Evaluation (SemEval-2023)

View full text Add to dashboard Cite

show abstract

“…A study by He et al (2021) demonstrated that adapterbased tuning exhibits enhanced stability and generalization capabilities by virtue of being less sensitive to learning rates than traditional fine-tuning methods. While incorporating task adaptation techniques, such as TAPT, has been shown to match or even improve performance over FFT in lowresource setups, Kim et al (2021) noted an interesting caveat: the benefits of integrating TAPT with adapters tend to taper off as the amount of data increases.…”

Section: Related Workmentioning

confidence: 99%

Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings

Jukić,

Snajder

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have ignited a surge in demand for effective finetuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning. 1

show abstract

“…Thereby we aim to bring the models closer to the target domains of the task and induce increased task performance compared to the base models. We also apply adapter-based (Houlsby et al, 2019;Pfeiffer et al, 2020) intermediate pretraining to compare full TAPT against to a more parameter efficient approach (Kim et al, 2021).…”

Section: Domain Adaption and Generalizationmentioning

confidence: 99%

Multi-Source (Pre-)Training for Cross-Domain Measurement, Unit and Context Extraction

Li¹,

Martschat²,

Ponzetto³

2023

The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks

View full text Add to dashboard Cite

We present a cross-domain approach for automated measurement and context extraction based on pre-trained language models. We construct a multi-source, multi-domain corpus and train an end-to-end extraction pipeline. We then apply multi-source task-adaptive pre-training and fine-tuning to benchmark the cross-domain generalization capability of our model. Further, we conceptualize and apply a task-specific error analysis and derive insights for future work. Our results suggest that multi-source training leads to the best overall results, while singlesource training yields the best results for the respective individual domain. While our setup is successful at extracting quantity values and units, more research is needed to improve the extraction of contextual entities. We make the cross-domain corpus used in this work available online 1 .

show abstract

Revisiting Pretraining with Adapters

Cited by 7 publications

References 15 publications

SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification

SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification

Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings

Multi-Source (Pre-)Training for Cross-Domain Measurement, Unit and Context Extraction

Contact Info

Product

Resources

About