FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models

Abstract

The task of learning from only a few examples (called a few-shot setting) is of key importance and relevance to a real-world setting. For question answering (QA), the current state-of-the-art pre-trained models typically need fine-tuning on tens of thousands of examples to obtain good results. Their performance degrades significantly in a few-shot setting (< 100 examples). To address this, we propose a simple fine-tuning framework that leverages pre-trained text-to-text models and is directly aligned with their pre-training framework. Specifically, we construct the input as a concatenation of the question, a mask token representing the answer span and a context. Given this input, the model is fine-tuned using the same objective as that of its pre-training objective. Through experimental studies on various few-shot configurations, we show that this formulation leads to significant gains on multiple QA benchmarks (an absolute gain of 34.2 F1 points on average when there are only 16 training examples). The gains extend further when used with larger models (Eg:-72.3 F1 on SQuAD using BART-large with only 32 examples) and translate well to a multilingual setting . On the multilingual TydiQA benchmark, our model outperforms the XLM-Roberta-large by an absolute margin of upto 40 F1 points and an average of 33 F1 points in a few-shot setting (<= 64 training examples). We conduct detailed ablation studies to analyze factors contributing to these gains.

Towards More Realistic Generation of Information-Seeking Conversations

“…In particular, we highlight target answer a t as rationale span in the passage c using a special token, "<hl>", following Gu et al (2021). In addition, we adopt a mask prediction scheme that aligns its objective with that of the pre-training phase, shown to be sample efficient in prior work (Chada and Natarajan, 2021).…”

“…They are also concatenated with the <sep> token to represent the B. The masked question prediction scheme is inspired by Chada and Natarajan (2021) and we find the scheme is more sample efficient in our preliminary experiment. We train both CQG models for 10 epochs with 16 for batch size, 3e-5 for lr, 0.1 for lr warming up, and 0.01 for weight decay on 2 32GB V100 GPUs.…”

Reasoning Circuits: Few-shot Multihop Question Generation with Structured Rationales

“…We use the standard encoder-decoder objective of maximizing the log likelihood of text in the ground truth target. For creating input and output prompts, we closely follow the templates proposed by Chada and Natarajan (2021) where prompts are aligned to the format used during the MLM pretraining.…”

Few-Shot Multihop Question Answering over Knowledge Base

“…Since the expensiveness of constructing the annotated datasets, several works have been focused on few-shot learning for KBQA task. Chada et al (2021) proposed a simple fine-tuning framework that regards the query path generation as a text-to-text task [40]. By leveraging a pre-trained sequence-to-sequence models, their method outperforms many state-of-art models with an average margin of 34.2 F1 points on various few-shot settings of multiple QA benchmarks.…”