Learning from Task Descriptions

Abstract

Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task. Moreover, the dataset's structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers. 1

Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?

“…show that models can take in rules and perform soft reasoning on them. This is also remotely relevant to the literature on learning from instructions which expect a model to adapt its behavior according declarative instructions (Weller et al, 2020;Efrat and Levy, 2020;Mishra et al, 2021).…”

Learning to Generate Task-Specific Adapters from Task Description

“…For instance, in the ZEST dataset (Weller et al, 2020), a train task description can be "Are mountain bikes allowed at this national park? ", while D contains twenty paragraphs for different national parks and twenty corresponding answers.…”

“…We use two datasets that fit our setup. The first one is Zero-shot Learning from Task Descriptions dataset (ZEST, Weller et al 2020), which formulates task descriptions as generalized questions, and provides multiple source-target examples for each question. The performance is evaluated with a novel metric: "Competence@K", along with mean F1 score.…”

“…Pre-trained text-to-text models (Raffel et al, 2020;Lewis et al, 2020) provide a unified formulation and off-the-shelf weights for a variety of NLP tasks, such as question answering (Khashabi et al, 2020) and commonsense reasoning (Bosselut et al, 2019). In addition to their strong performance, text-totext models naturally support generalizing to novel tasks, by incorporating task description as part of the source sequence and fine-tuning the model with (source, target) examples (Weller et al, 2020). At inference time, the model is required to perform Figure 1: Instead of learning from (source, target) examples, in this paper we study the problem of learning from task descriptions (Weller et al, 2020).…”

“…In addition to their strong performance, text-totext models naturally support generalizing to novel tasks, by incorporating task description as part of the source sequence and fine-tuning the model with (source, target) examples (Weller et al, 2020). At inference time, the model is required to perform Figure 1: Instead of learning from (source, target) examples, in this paper we study the problem of learning from task descriptions (Weller et al, 2020). The train set contains M tasks, and the i-th task contains N i examples of (s, t) pairs in text format.…”