RACE: Large-scale ReAding Comprehension Dataset From Examinations

Lai, Guokun; Xie, Qizhe; Liu, Hanxiao; Yang, Yiming; Hovy, Eduard

doi:10.48550/arxiv.1704.04683

Cited by 138 publications

(133 citation statements)

References 7 publications

Supporting

Mentioning

133

Contrasting

Order By: Relevance

“…Unlike the previous three datasets, Nar-rativeQA [35] does not restrict the answers to be the span of texts in the articles, therefore, it can be used as an answer-abstraction QG dataset. Race [38], McTest [64], OpenbookQA [49], and ARC [15] are commonly used multi-choice QG datasets. BoolQA [14] is a typical boolean QG dataset, gathered from Google search engine.…”

Section: Related Work 21 Question Generationmentioning

confidence: 99%

“…• Answer-extraction QG: SQuADv1.1 [58] • Answer-abstraction QG: NarrativeQA [35] • Multi-choice QG: RACE [38], McTest [64], OpenbookQA [49], ARC-easy, ARC-hard [15] • Boolean QG: BoolQA [14] We assume datasets arrive in the following order: "McTest→SQuAD →RACE→NarrativeQA→Arc-easy→Arc-hard→OpenbookQA →BoolQA", which corresponds to the exact release dates of these datasets in the real world. Details on dataset characteristics, statistics, and splitting strategies are in Appendix A.1.…”

Section: Experiments 51 Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Unified Question Generation with Continual Lifelong Learning

Yuan,

Yin,

et al. 2022

Preprint

View full text Add to dashboard Cite

Question Generation (QG), as a challenging Natural Language Processing task, aims at generating questions based on given answers and context. Existing QG methods mainly focus on building or training models for specific QG datasets. These works are subject to two major limitations: (1) They are dedicated to specific QG formats (e.g., answer-extraction or multi-choice QG), therefore, if we want to address a new format of QG, a re-design of the QG model is required. (2) Optimal performance is only achieved on the dataset they were just trained on. As a result, we have to train and keep various QG models for different QG datasets, which is resource-intensive and ungeneralizable.To solve the problems, we propose a model named Unified-QG based on lifelong learning techniques, which can continually learn QG tasks across different datasets and formats. Specifically, we first build a format-convert encoding to transform different kinds of QG formats into a unified representation. Then, a method named STRIDER (SimilariT y RegularI zed Difficult Example Replay) is built to alleviate catastrophic forgetting in continual QG learning. Extensive experiments were conducted on 8 QG datasets across 4 QG formats (answer-extraction, answer-abstraction, multi-choice, and boolean QG) to demonstrate the effectiveness of our approach. Experimental results demonstrate that our Unified-QG can effectively and continually adapt to QG tasks when datasets and formats vary. In addition, we verify the ability of a single trained Unified-QG model in improving 8 Question Answering (QA) systems' performance through generating synthetic QA data.

show abstract

Section: Related Work 21 Question Generationmentioning

confidence: 99%

Section: Experiments 51 Datasetsmentioning

confidence: 99%

Unified Question Generation with Continual Lifelong Learning

Yuan,

Yin,

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Multiple-choice datasets overcome the evaluation difficulty in abstractive datasets by simply asking the reader to select the correct answer from the candidate options. Representative datasets, such as RACE [19] and DREAM [33], utilize exam questions collected from standard English proficiency tests, which are generated by language teachers to evaluate a variety of language capabilities of non-native English speakers.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, the C 3 dataset [34] was released, which contains both free-form questions and multiple-choice answer options. C 3 is collected using the exam questions for Chinese-as-a-second-language tests and consists of two sub-datasets, C 3 D focusing on normal documents and C 3 M on dialogues, which can be viewed as the Chinese counterparts of RACE [19] and DREAM [33] respectively.…”

Section: Related Workmentioning

confidence: 99%

“…Machine reading comprehension (MRC) is one of the fundamental tasks in natural language understanding, which requires a machine to read a document to correctly answer questions based on the context. MRC has attracted significant efforts from both academia and industry with continuous development of MRC datasets [15,23,19,26,28,33], which also keeps pushing the frontier of MRC models and learning algorithms [11,22,20,3] to eventually bridge the gap between AI systems and human readers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension

Xu¹,

Yi-chen²,

Yi³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present Native Chinese Reader (NCR), a new machine reading comprehension (MRC) dataset with particularly long articles in both modern and classical Chinese. NCR is collected from the exam questions for the Chinese course in China's high schools, which are designed to evaluate the language proficiency of native Chinese youth. Existing Chinese MRC datasets are either domain-specific or focusing on short contexts of a few hundreds of characters in modern Chinese only. By contrast, NCR contains 8390 documents with an average length of 1024 characters covering a wide range of Chinese writing styles, including modern articles, classical literature and classical poetry. A total of 20477 questions on these documents also require strong reasoning abilities and common sense to figure out the correct answers. We implemented multiple baseline models using popular Chinese pretrained models and additionally launched an online competition using our dataset to examine the limit of current methods. The best model achieves 59% test accuracy while human evaluation shows an average accuracy of 79%, which indicates a significant performance gap between current MRC models and native Chinese speakers. We release the dataset at https://sites.google.com/ view/native-chinese-reader/.

show abstract

Improving Question Answering by Commonsense-Based Pre-training

Zhong

Tang

Duan

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Although neural network approaches achieve remarkable success on a variety of NLP tasks, many of them struggle to answer questions that require commonsense knowledge. We believe the main reason is the lack of commonsense connections between concepts. To remedy this, we provide a simple and effective method that leverages external commonsense knowledge base such as Con-ceptNet. We pre-train direct and indirect relational functions between concepts, and show that these pre-trained functions could be easily added to existing neural network models. Results show that incorporating commonsense-based function improves the baseline on three question answering tasks that require commonsense reasoning. Further analysis shows that our system discovers and leverages useful evidence from an external commonsense knowledge base, which is missing in existing neural network models and help derive the correct answer.

show abstract

RACE: Large-scale ReAding Comprehension Dataset From Examinations

Cited by 138 publications

References 7 publications

Unified Question Generation with Continual Lifelong Learning

Unified Question Generation with Continual Lifelong Learning

Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension

Improving Question Answering by Commonsense-Based Pre-training

Contact Info

Product

Resources

About