UnitedQA: A Hybrid Approach for Open Domain Question Answering

Cheng, Hao; Shen, Yulong; Liu, Xiaodong; He, Pengcheng; Chen, Weizhu; Gao, Jianfeng

doi:10.18653/v1/2021.acl-long.240

Cited by 19 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We hypothesize this may be caused by ELECTRA pre-training method, which shows strong performance through variety of tasks and we further show that it is also due to training and inference with large input size of 128 passages and better objective (discussed in Section 4.2 and Appendix G). Only system that matches the performance of our extractive reader is the concurrent work on UnitedQA-E (Cheng et al, 2021), which uses advanced regularization and Har-dEM techniques. We note that these are orthogonal to our approach and could potentially lead to further improvements.…”

Section: Results and Analysismentioning

confidence: 99%

“…Extractive BM25+BERT (Mao et al, 2020) 37.7 60.1 110M Hard EM (Min et al, 2019a) 28.1 50.9 110M Path Retriever (Asai et al, 2020) 32.6 -447M Graph Retriever (Min et al, 2019b) 34.5 56.0 110M ORQA 33.3 45.0 220M REALM (Guu et al, 2020) 40.4 -660M ProQA (Xiong et al, 2021) 34.3 -220M DPR 41.5 56.8 220M RDR (Yang and Seo, 2020) 42.1 57.0 110M GAR+DPR (Mao et al, 2020) 43.8 -626M ColBERT (Khattab et al, 2020) 48.2 63.2 − 440M RIDER (GAR+DPR) (Mao et al, 2021) 48.3 -626M UnitedQA-E (Cheng et al, 2021) 51.8 68.9 440M Generative BM25+SSG (Mao et al, 2020) 35.3 58.6 406M T51.1+SSM 35.2 61.6 11B RAG 44.5 56.8 516M DPR+SSG 42.2 -516M FiD-base (Izacard and Grave, 2021) 48.2 65.0 333M FiD-large (Izacard and Grave, 2021) 51.4 67.6 848M FiD-large++ 54 Finally, we find that our R2-D2 system with 21M passages corpus is competitive even with FiD++, which uses DPR retriever improved via knowledge distillation, and 26M passage corpus, which also includes lists. Additionally, we evaluate our model with a better retrieval model (HN-DPR) based on the DPR checkpoint where hard negatives are mined using the retrieval model itself 11 .…”

Section: Methods Nq Tq #θmentioning

confidence: 99%

“…the extractive reader. Concurrently with our work, Cheng et al (2021) proposed hard voting ensembling scheme to combine the reader predictions. Firstly, each model from an ensemble produces its best prediction, then the votes for identical predictions are combined, omitting the scores produced by the individual models.…”

Section: Related Workmentioning

confidence: 96%

See 2 more Smart Citations

R2-D2: A Modular Baseline for Open-Domain Question Answering

Fajcik¹,

Docekal²,

Ondrej³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

This work presents a novel four-stage opendomain QA pipeline R2-D2 (RANK TWICE, READ TWICE). The pipeline is composed of a retriever, passage reranker, extractive reader, generative reader and a mechanism that aggregates the final prediction from all system's components. We demonstrate its strength across three open-domain QA datasets: Natu-ralQuestions, TriviaQA and EfficientQA, surpassing state-of-the-art on the first two. Our analysis demonstrates that: (i) combining extractive and generative reader yields absolute improvements up to 5 exact match and it is at least twice as effective as the posterior averaging ensemble of the same models with different parameters, (ii) the extractive reader with fewer parameters can match the performance of the generative reader on extractive QA datasets 1 .

show abstract

Section: Results and Analysismentioning

confidence: 99%

Section: Methods Nq Tq #θmentioning

confidence: 99%

See 1 more Smart Citation

R2-D2: A Modular Baseline for Open-Domain Question Answering

Fajcik¹,

Docekal²,

Ondrej³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

show abstract

“…This approach was also applied in RocketQA . UnitedQA (Cheng et al, 2021) and R2D2 (Fajcik et al, 2021) combine results from an ensemble of extractive and generative readers, whereas PAQ directly retrieves possible answers with an FiD fallback.…”

Section: E2e Optimization Of Nralmmentioning

confidence: 99%

You Only Need One Model for Open-domain Question Answering

Lee¹,

Kedia²,

Lee³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent works for Open-domain Question Answering refer to an external knowledge base using a retriever model, optionally rerank the passages with a separate reranker model and generate an answer using an another reader model. Despite performing related tasks, the models have separate parameters and are weakly-coupled during training. In this work, we propose casting the retriever and the reranker as hard-attention mechanisms applied sequentially within the transformer architecture and feeding the resulting computed representations to the reader. In this singular model architecture the hidden representations are progressively refined from the retriever to the reranker to the reader, which is more efficient use of model capacity and also leads to better gradient flow when we train it in an end-to-end manner. We also propose a pretraining methodology to effectively train this architecture. We evaluate our model on Natural Questions and TriviaQA open datasets and for a fixed parameter budget, our model outperforms the previous state-of-the-art model by 1.0 and 0.7 exact match scores.

show abstract

“…All these benchmarks either assume that each question has only one answer with several alternative surface forms, or only require a system to predict one valid answer. A typical question answering system is a pipeline as follows: an efficient retriever retrieves relevant passages using sparse (Mao et al, 2021; or dense Xiong et al, 2021;Izacard and Grave, 2021a;Khattab et al, 2021) representations; an optional passage reranker (Asadi and Lin, 2013;Nogueira and Cho, 2019;Nogueira et al, 2020) further narrows down the evidence; an extractive or generative reader (Izacard and Grave, 2021b;Cheng et al, 2021) predicts an answer conditioned on retrieved or top-ranking passages. Nearly all previous work focused on locating passages covering at least one answer, or tried to predict one answer precisely.…”

Section: Related Workmentioning

confidence: 99%

Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework

Shao¹,

Huang²

2021

Preprint

View full text Add to dashboard Cite

Open domain questions are likely to be openended and ambiguous, leading to multiple valid answers. Existing approaches typically adopt the rerank-then-read framework, where a reader reads top-ranking evidence to predict answers. According to our empirical analyses, this framework is faced with three problems: to leverage the power of a large reader, the reranker is forced to select only a few relevant passages that cover diverse answers, which is non-trivial due to unknown effect on the reader's performance; the small reading budget also prevents the reader from making use of valuable retrieved evidence filtered out by the reranker; besides, as the reader generates predictions all at once based on all selected evidence, it may learn pathological dependencies among answers, i.e., whether to predict an answer may also depend on evidence of the other answers. To avoid these problems, we propose to tackle multi-answer open-domain questions with a recall-then-verify framework, which separates the reasoning process of each answer so that we can make better use of retrieved evidence while also leveraging the power of large models under the same memory constraint. Our framework achieves new stateof-the-art results on two multi-answer datasets, and predicts significantly more gold answers than a rerank-then-read system with an oracle reranker.

show abstract

UnitedQA: A Hybrid Approach for Open Domain Question Answering

Cited by 19 publications

References 27 publications

R2-D2: A Modular Baseline for Open-Domain Question Answering

R2-D2: A Modular Baseline for Open-Domain Question Answering

You Only Need One Model for Open-domain Question Answering

Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework

Contact Info

Product

Resources

About