Multitask Prompted Training Enables Zero-Shot Task Generalization

Abstract

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language model training . Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping general natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts using varying natural language. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. We fine-tune a pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16× its size. Further, our approach attains strong performance on a subset of tasks from the BIG-Bench benchmark, outperforming models up to 6× its size. All prompts and trained models are available at github.com/bigscience-workshop/promptsource/ and huggingface.co/bigscience/T0pp.

Using natural language prompts for machine translation

“…Large language models can generalize to these unseen instructions, obtaining reasonable performance in a wide variety of tasks. Moreover, recent works Sanh et al, 2021) have shown that we can improve the performance of this instructionfollowing behavior by fine-tuning on a multi-task mixture using natural language descriptions of the tasks, which mirrors closely the results from the multilingual MT literature.…”

“…In this formulation, we could formulate our prompts as Translate to {language_name}: {input_slot}. and Sanh et al (2021) have shown this ability to follow natural language instructions can be improved by finetuning the model on a diverse mixture of tasks. All of these works have typically focused on large, English-centric language models.…”

CheckDST: Measuring Real-World Generalization of Dialogue State Tracking Performance

“…Motivated by this idea, we combine approaches from SOLOIST, MUPPET, and T0 for PrefineDST in an attempt to train a robust DST model through prefinetuning (Peng et al, 2020a;Aghajanyan et al, 2021;Sanh et al, 2021). We choose prefinetuning tasks based on their intuitive potential for improving on qualities measured by CheckDST and uniformly format these non-target datasets as text-totext generation tasks with the help of instruction prompts.…”

“…The most similar work to PrefineDST is MUPPET, a BART model prefinetuned on more than 50 heterogeneous tasks via additional layers that accommodate different task structures (Aghajanyan et al, 2021). We adapt the multitasking approach by Sanh et al (2021) and to remove the additional layers used by MUPPET.…”

“…Following Sanh et al (2021), we do not adjust the sampling rate based on each the sample size of each task that we multitask with during prefinetuning. Since all tasks are formatted as a text-to-text generation task, we do not need any additional layers as was needed for MUPPET nor form homogeneous batches that contain samples only from the same task.…”