NewsQA: A Machine Comprehension Dataset

Abstract

We present NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text in the articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. Analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (13.3% F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available online.

Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability

“…NewsQA (Trischler et al, 2016): we randomly chose questions that satisfied the following conditions:…”

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)

“…NewsQA The NewsQA dataset (Trischler et al, 2017) 3 contains 100k answerable questions from a total of 120k questions. The dataset is built from CNN news stories that were originally collected by Hermann et al (2015).…”

“…A thorough analysis by Chen et al (2016), however, revealed that the DailyMail/CNN was too easy and still quite noisy. New datasets were constructed to eliminate these problems including SQuAD (Rajpurkar et al, 2016), NewsQA (Trischler et al, 2017) and MsMARCO (Nguyen et al, 2016).…”

“…The creation of large-scale, extractive QA datasets (Rajpurkar et al, 2016;Trischler et al, 2017;Nguyen et al, 2016) sparked research interest into the development of end-to-end neural QA systems. A typical neural architecture consists of an embedding-, encoding-, interaction-and answer layer (Wang and Jiang, 2017;Yu et al, 2017;Xiong et al, 2017;Seo et al, 2017;Yang et al, 2017;.…”

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

“…Datasets with natural language questions include MCTest (Richardson et al, 2013), SQuAD (Rajpurkar et al, 2016), and NewsQA (Trischler et al, 2016). MCTest is limited in scale with only 2640 multiple choice questions.…”