Span Selection Pre-training for Question Answering

Abstract

BERT (Bidirectional Encoder Representations from Transformers) and related pre-trained Transformers have provided large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA). BERT is pretrained on two auxiliary tasks: Masked Language Model and Next Sentence Prediction. In this paper we introduce a new pre-training task inspired by reading comprehension to better align the pre-training from memorization to understanding. Span Selection Pre-Training (SSPT) poses cloze-like training instances, but rather than draw the answer from the model's parameters, it is selected from a relevant passage. We find significant and consistent improvements over both BERT BASE and BERT LARGE on multiple Machine Reading Comprehension (MRC) datasets. Specifically, our proposed model has strong empirical evidence as it obtains SOTA results on Natural Questions, a new benchmark MRC dataset, outperforming BERT LARGE by 3 F1 points on short answer prediction. We also show significant impact in HotpotQA, improving answer prediction F1 by 4 points and supporting fact prediction F1 by 1 point and outperforming the previous best system. Moreover, we show that our pre-training approach is particularly effective when training data is limited, improving the learning curve by a large amount.

ConVEx: Data-Efficient and Few-Shot Slot Labeling

“…Reddit has been shown to provide natural conversational English data for learning semantic representations that work well in downstream tasks related to dialog and conversation (Al-Rfou et al, 2016;Cer et al, 2018;Henderson et al, 2019bCoope et al, 2020). Therefore, following 1 The pairwise cloze task has been inspired by the recent span selection objective applied to extractive QA by Glass et al (2020): they create examples emulating extractive QA pairs with long passages and short question sentences. Another similar approach to extractive QA has been proposed by Ram et al (2021).…”

“…However, we detect several gaps with the existing setup, and set to address them in this work. First, recent work in NLP has validated that a stronger alignment between a pretraining task and an end task can yield performance gains for tasks such as extractive question answering (Glass et al, 2020) and paraphrase and translation (Lewis et al, 2020). We ask whether it is possible to design a pretraining task which is more suitable for slot labeling in conversational applications.…”

Is Graph Structure Necessary for Multi-hop Question Answering?

“…Baseline 10.83 40.16 QFE (Nishida et al, 2019) 34.63 59.61 DFGN (Qiu et al, 2019) 33.62 59.82 TAP2 (Glass et al, 2019) 39.77 69.12 HGN (Fang et al, 2019) 43.57 71.03 SAE (Tu et al, 2019a) 45 eral extra modules in the graph fusion block, including query-entity attention, query update mechanism, and weak supervision. Prediction Layer.…”

CFO: A Framework for Building Production NLP Systems

“…Table 1 shows that this strategy can provide an absolute improvement of 2.5% over a model that starts with just the default BERT language model. 12 See http://www.ibm.biz/confidence_ thresholding for more on choosing business specific thresholds 13 We only use 1 P100 GPU or 8 CPU threads in latency experiments Pre-Training EM F1 BERT (Devlin et al, 2018) We also employ (Glass et al, 2019)'s approach to using an unsupervised auxilary task that is better aligned to our final task (i.e. MRC) than the default Masked Language Model and Next Sentence Prediction used in (Devlin et al, 2018) to pre-train the BERT models.…”