Muppet: Massive Multi-task Representations with Pre-Finetuning

Aghajanyan, Armen; Gupta, Anchit; Shrivastava, Akshat; Chen, Xilun; Zettlemoyer, Luke; Gupta, Sonal

doi:10.18653/v1/2021.emnlp-main.468

Cited by 138 publications

(129 citation statements)

References 47 publications

Supporting

Mentioning

126

Contrasting

Order By: Relevance

“…QQP) is of interest. This finding, along with prior work demonstrating the effectiveness of multitask training (Wang et al, 2019;Liu et al, 2019;Aghajanyan et al, 2021), motivates our next question: if we only care about a single task, should we still use a multitask objective?…”

Section: Multitask Training As An Auxiliary Pruning Objectivementioning

confidence: 88%

“…In addition to the benefits described above, we aim for the best of both worlds: a substantially pruned model that also performs well on multiple tasks. We also ask whether individual task performances can be improved by leveraging data from the other tasks, which is a strategy employed by general-purpose language modeling (Liu et al, 2019;Aghajanyan et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Pruning Pretrained Encoders with a Multitask Objective

Xia¹,

Shin²

2021

Preprint

View full text Add to dashboard Cite

The sizes of pretrained language models make them challenging and expensive to use when there are multiple desired downstream tasks. In this work, we adopt recent strategies for model pruning during finetuning to explore the question of whether it is possible to prune a single encoder so that it can be used for multiple tasks. We allocate a fixed parameter budget and compare pruning a single model with a multitask objective against the best ensemble of single-task models. We find that under two pruning strategies (element-wise and rank pruning), the approach with the multitask objective outperforms training models separately when averaged across all tasks, and it is competitive on each individual one. Additional analysis finds that using a multitask objective during pruning can also be an effective method for reducing model sizes for low-resource tasks.

show abstract

Section: Multitask Training As An Auxiliary Pruning Objectivementioning

confidence: 88%

Section: Introductionmentioning

confidence: 99%

Pruning Pretrained Encoders with a Multitask Objective

Xia¹,

Shin²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…-We propose EXMIX ( §2): a collection of 107 supervised NLP tasks for Extreme Multi-task Scaling, formatted for encoder-decoder training. EXMIX has approximately twice as many tasks as the largest prior study to date (Aghajanyan et al, 2021), totaling 18M labeled examples across diverse task families.…”

Section: Ext5mentioning

confidence: 99%

“…For the first time, we explore and propose Extreme Multi-task Scaling -a new paradigm for multitask pre-training. Compared to the largest prior work (Aghajanyan et al, 2021), our study doubles the number of tasks and focuses on multi-task pre-training rather than fine-tuning, which enables a direct comparison to standard pre-training. Our proposal is based on the insight that despite negative transfer being common during fine-tuning, a massive and diverse collection of pre-training tasks is generally preferable to an expensive search for the best combination of pre-training tasks.…”

Section: Introductionmentioning

confidence: 99%

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

Aribandi¹,

Tay²,

Schuster³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces EXMIX (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using EXMIX, we study the effect of multi-task pre-training at the largest scale to date, and analyze cotraining transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose EXT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised EXMIX. Via extensive experiments, we show that EXT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of EXMIX. EXT5 also significantly improves sample efficiency while pre-training. * Google AI Resident. † Equal contribution. Sebastian is now at Google Research. Sanket returned to CMU.

show abstract

“…Large models trained on unsupervised objectives on massive text corpora can be finetuned on new tasks and typically perform better than models trained from scratch (Devlin et al, 2018;Radford et al, 2018;Conneau et al, 2019). Moreover, learning from multiple tasks has been shown to be helpful for generalization at all stages of training: during pretraining (Aribandi et al, 2021), as a special training stage before fine-tuning (Aghajanyan et al, 2021), and even during fine-tuning (Tang et al, 2020). In all cases, one attains stronger positive transfer by training on larger number of tasks, intensifying the need for a flexible way to handle an arbitrary number of tasks.…”

Section: Introductionmentioning

confidence: 99%

Using natural language prompts for machine translation

García¹,

Fırat²

2022

Preprint

View full text Add to dashboard Cite

We explore the use of natural language prompts for controlling various aspects of the outputs generated by machine translation models. We demonstrate that natural language prompts allow us to influence properties like formality or specific dialect of the output. We show that using language names to control the output language of multilingual translation models enables positive transfer for unseen language pairs. This unlocks the ability to translate into languages not seen during finetuning by using their English names. We investigate how scale, number of pre-training steps, number of languages in fine-tuning, and language similarity affect this phenomenon.

show abstract

Muppet: Massive Multi-task Representations with Pre-Finetuning

Cited by 138 publications

References 47 publications

Pruning Pretrained Encoders with a Multitask Objective

Pruning Pretrained Encoders with a Multitask Objective

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

Using natural language prompts for machine translation

Contact Info

Product

Resources

About