The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

Abstract

We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 . Through careful ablation studies on the Flan Collection of tasks and methods, we tease apart the effect of design decisions which enable Flan-T5 to outperform prior work by 3-17%+ across evaluation settings. We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, and chain-of-thought) actually yields stronger (2%+) performance in all settings. In further experiments, we show Flan-T5 requires less finetuning to converge higher and faster than T5 on single downstream tasks-motivating instruction-tuned models as more computationally-efficient starting checkpoints for new tasks. Finally, to accelerate research on instruction tuning, we make the Flan 2022 collection of datasets, templates, and methods publicly available. 1

Applications of transformer-based language models in bioinformatics: a survey

“…In this approach, (instruction, output) pairs are collected from existing annotated natural language datasets by using templates to transform text-label pairs to (instruction, output) pairs. Datasets such as Flan (Longpre et al, 2023) and P3 (Sanh et al, 2021) are constructed based on the data integration strategy.…”

“…Flan 2021 (Longpre et al, 2023) is an English instruction dataset constructed by transforming 62 widely-used NLP benchmarks (e.g., SST-2, SNLI, AG News, MultiRC) into language inputoutput pairs. Each instance in the Flan 2021 has "input" and "target" components.…”

“…Flan-T5 (11B) is is a large language model initialized with T5 (11B) (Raffel et al, 2019), and then fine-tuned on the FLAN dataset (Longpre et al, 2023). The FLAN dataset is a collection of (instruction, pairs) pairs, constructed from 62 datasets of 12 NLP tasks (e.g., natural language inference, commonsense reasoning, paraphrase generation) by filling templates with various instructions under a unified task formalization.…”

“…OPT-IML (175B) (Iyer et al, 2022) is a large language model trained by fine-tuning the OPT (175B) (Zhang et al, 2022a) model on the constructed Instruction Meta-Learning (IML) dataset, which consists of over 1500 NLP tasks from 8 publicly available benchmarks such as PromptSource (Bach et al, 2022), FLAN (Longpre et al, 2023), and Super-NaturalInstructions (Wang et al, 2022d). After fine-tuning, OPT-IML outperforms OPT across all benchmarks.…”

AugGPT: Leveraging ChatGPT for Text Data Augmentation

“…In recent years, NLP communities have witnessed the rise of very large language models such as GPT-3 (175B parameters) [8], PaLM (540B parameters) [49], Bloom (176B parameters) [50], OPT (up to 175B parameters) [51], and the FLAN series (FLAN has 137B parameters) [52]. At their core, these large language models are transformer models inspired by BERT and GPT, albeit at a much larger scale.…”