UL2: Unifying Language Learning Paradigms

Abstract

Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -two concepts that are commonly conflated. Next, we present a generalized and unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 and/or GPT-like models across multiple diverse setups. Finally, by scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised NLP tasks ranging from language generation (with automated and human evaluation), language understanding, text classification, question answering, commonsense reasoning, long text reasoning, structured knowledge grounding and information retrieval. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. We release Flax-based T5X model checkpoints for the 20B model at https: //github.com/google-research/google-research/tree/master/ul2. * Yi and Mostafa are co-leads of this project and are denoted with * . denotes technical research contributors. denotes data & infrastructure contributions. denotes advising contributions. Don, denoted with is the senior most last author.

Recitation-Augmented Language Models

“…UL2 UL2 (Unifying Language Learning, Tay et al 2022a) is an encoder-decoder model trained on a mixture of denoising tasks in a unified framework. In this paper, since we mainly focus on the in-context learning ability of language models, we use UL2-20B in the S-Denoiser mode (i.e., pre-trained with the prefix language modeling) 1 .…”

“…Model weights The model weights of two LLMs used in our experiments, i.e., UL2-20B (Tay et al, 2022a) and OPT-30B , are publicly released through GCP bucket (gs: //scenic-bucket/ul2) and Github (https://github.com/facebookresearch/ metaseq), respectively.…”

“…Experiments on three large language models (PaLM (Chowdhery et al, 2022), UL2 (Tay et al, 2022a), and OPT ) show that a recite-and-answer scheme can improve performance on various types of CBQA tasks, including Wikipedia-based single-hop QA (Natural Questions, Kwiatkowski et al 2019), trivia questions (TriviaQA, Joshi et al 2017), andWikipedia-based multi-hop QA (HotpotQA, Yang et al 2018).…”

Can Foundation Models Help Us Achieve Perfect Secrecy?

“…A promise of FMs is their adaptability, enabling the use a single FM (order 100M-100B parameters) for multiple tasks versus a unique model per personal task in FL. Our work shows there is work to be done to achieve this promise, though we see the development of more versatile FMs over time [77,69,37]. For small FL models (100K parameters), the multiplicative factor is insignificant, yet model quality improves using more parameters, or training for longer over more data [32].…”

Stateful Memory-Augmented Transformers for Dialogue Modeling

“…The third is the extension to other Transformer models. We focused on Transformer encoderdecoder models, because the recent large encoderdecoder model UL2 (Tay et al, 2022) with 20B parameters has shown better zero-shot performance than GPT3 175B, suggesting the effectiveness of encoder-decoder models. In the future, we will extend our approach to large Transformer encoderdecoders such as T5 and UL2.…”