Task Transfer and Domain Adaptation for Zero-Shot Question Answering

Abstract

Pretrained language models have shown success in various areas of natural language processing, including reading comprehension tasks. However, when applying machine learning methods to new domains, labeled data may not always be available. To address this, we use supervised pretraining on source-domain data to reduce sample complexity on domainspecific downstream tasks. We evaluate zeroshot performance on domain-specific reading comprehension tasks by combining task transfer with domain adaptation to fine-tune a pretrained model with no labelled data from the target task. Our approach outperforms Domain-Adaptive Pretraining on downstream domain-specific reading comprehension tasks in 3 out of 4 domains.

Compositional Zero-Shot Domain Transfer with Text-to-Text Models

“…The definition of 'zero-shot' in this paper follows recent studies(Pan et al, 2022;Zhao et al, 2022), and is similar to unsupervised domain adaptation, as discussed in §2. Another similar usage of 'zero-shot' is found in cross-lingual setups where no task labels are accessible in the target test language but labels in the same task are available in a source language.…”

“…https://beta.openai.com/examples/ default-tldr-summary 10 These are still zero-shot baselines as they do not use indomain task examples.11 This baseline category is similar to contemporaneous work(Pan et al, 2022) where domain-task transfer is achieved through sequential in-domain off-task training followed by general-domain in-task training. Here we do not use indomain task data of any kind.…”