Quasar: Datasets for Question Answering by Search and Reading

Dhingra, Bhuwan; Mazaitis, Kathryn; Cohen, William W.

doi:10.48550/arxiv.1707.03904

Cited by 46 publications

(77 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The encoder uses self-attention to conditionally encode the context with the query and the decoder allows conditional generation of outputs that are not necessarily present in the input. To scale question answering to reason over large knowledge-sources such as Wikipedia, task formulations typically retrieve textspans from a corpus to condition answer generation (Chen et al, 2017;Dhingra et al, 2017). However, several challenges encountered in NLDBs preclude direct application of these techniques: Scale To scale neural reasoning to databases of non-trivial size, it would not be feasible to encode the entire database as input to the transformer.…”

Section: Challengesmentioning

confidence: 99%

“…Machines have surpassed human performance on the well-known SQUaD task (Rajpurkar et al, 2016) where models extract answer spans from a short passage of text. The subsequent body of work has further considered incorporating retrieval from large corpora such as Wikipedia (Dhingra et al, 2017;Joshi et al, 2017;Kwiatkowski et al, 2019) to identify relevant information, conditioning answer generation (Chen -Sarah works in a hospital in NY as a doctor.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Database reasoning over text

Thorne¹,

Yazdani²,

Saeidi³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Neural models have shown impressive performance gains in answering queries from natural language text. However, existing works are unable to support database queries, such as "List/Count all female athletes who were born in 20th century", which require reasoning over sets of relevant facts with operations such as join, filtering and aggregation. We show that while state-of-the-art transformer models perform very well for small databases, they exhibit limitations in processing noisy data, numerical operations, and queries that aggregate facts. We propose a modular architecture to answer these database-style queries over multiple spans from text and aggregating these at scale. We evaluate the architecture using WIKINLDB, 1 a novel dataset for exploring such queries. Our architecture scales to databases containing thousands of facts whereas contemporary models are limited by how many facts can be encoded. In direct comparison on small databases, our approach increases overall answer accuracy from 85% to 90%. On larger databases, our approach retains its accuracy whereas transformer baselines could not encode the context.

show abstract

Section: Challengesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Database reasoning over text

Thorne¹,

Yazdani²,

Saeidi³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…Previous open-book QA methods first filter a large corpus to a small set of relevant documents using information retrieval (Karpukhin et al, 2020;Robertson and Zaragoza, 2009). The document set then provides context for answering questions (Dhingra et al, 2017;Dunn et al, 2017;Joshi et al, 2017;Nguyen et al, 2016). Conversely, closed-book QA instead requires models to answer using only their implicit knowledge .…”

Section: Related Workmentioning

confidence: 99%

Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA

Ciosici¹,

Cecil²,

Hedges³

et al. 2021

Preprint

View full text Add to dashboard Cite

Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs.Since the questions are balanced, random performance should be ~50%. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook's content is not prerepresented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5's pre-training), yields at best minor improvement (56%), suggesting that the PTLM may not have "understood" the textbook (or perhaps misunderstood the questions). Performance is better (~60%) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).

show abstract

“…For the RC regime they use DRQA's document reader [7] while for the Open QA they utilize the PSPR model [27]. They experiment with different datasets (SQUAD [49] for RC and Quasar-T [10] for Open QA) for fine-tuning the models, as well as BioBert [26] embeddings to gain insights on the effect of the context length in this task.…”

Section: Task 7bmentioning

confidence: 99%

Results of the seventh edition of the BioASQ Challenge

Nentidis,

Bougiatiotis,

Krithara

et al. 2020

Preprint

View full text Add to dashboard Cite

The results of the seventh edition of the BioASQ challenge are presented in this paper. The aim of the BioASQ challenge is the promotion of systems and methodologies through the organization of a challenge on the tasks of large-scale biomedical semantic indexing and question answering. In total, 30 teams with more than 100 systems participated in the challenge this year. As in previous years, the best systems were able to outperform the strong baselines. This suggests that stateof-the-art systems are continuously improving, pushing the frontier of research.

show abstract

Quasar: Datasets for Question Answering by Search and Reading

Cited by 46 publications

References 0 publications

Database reasoning over text

Database reasoning over text

Perhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA

Results of the seventh edition of the BioASQ Challenge

Contact Info

Product

Resources

About