CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Abstract

Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CROSSFIT , a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CROSS-FIT and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from openaccess NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability. 1

Combining Modular Skills in Multitask Learning

“…In our paper, we efficiently achieve few-shot task adaptation by inferring the task-skill allocation matrix for new tasks and fine-tuning skill parameters, which were previously learned via multitask learning. In fact, Ye et al (2021) found that this pre-training routine is superior to meta-learning in CrossFit. A similar attempt to recompose modular knowledge learnt on previous tasks has been recently explored by Ostapenko et al (2021).…”

“…In addition, in the wake of the recent surge of interest in massively multitask few-shot NLP models (Min et al, 2021;Wei et al, 2021;Aribandi et al, 2021;Sanh et al, 2022;Karimi Mahabadi et al, 2021, inter alia), we also evaluate our latent-skill model on CrossFit (Ye et al, 2021). This benchmark recasts 160 NLP tasks (including QA, conditional text generation, classification, and other types such as regression) as textto-text generation problems.…”

“…In order to measure the benefits of a modular design for systematic generalisation to new tasks, we run a second set of experiments on CrossFit (Ye et al, 2021), a benchmark including 160 diverse natural language processing tasks sourced from Huggingface Datasets (Lhoest et al, 2021). The tasks in CrossFit are all converted into a unified 99.9 ± 0.2 99.9 ± 0.0 99.9 ± 0.0 99.9 ± 0.0 99.9 ± 0.0 99.9 ± 0.2 BSN-NPI-SCOPE Acc 85.7 ± 13.0 99.8 ± 0.3 99.6 ± 0.9 99.9 ± 0.2 99.9 ± 0.0 99.9 ± 0.2 BREAK-QDMR EM 4.8 ± 0.4 4.1 ± 0.5 1.9 ± 1.7 4.1 ± 0.9 3.8 ± 1.2 4.9 ± 0. text-to-text format inspired by Raffel et al (2020).…”

“…Hyper-parameter are tuned on the held-out set T dev . We adopt the partition 1 (called RANDOM) from Ye et al (2021), where |T train | = 120, |T dev | = |T eval | = 20, and tasks are split randomly. This is the most comprehensive partition and most suited for general-purpose models, as it includes all types of tasks.…”

“…Note that inYe et al (2021) the pre-trained model is BART Small and all parameters are fine-tuned. Hence, these results are not directly comparable.…”