DuReader_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications

Tang, Hongxuan; Li, Hongyu; Liu, Jing; Yu, Hong; Wu, Hua; Wang, Haifeng

doi:10.48550/arxiv.2004.11142

Cited by 2 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Text Summarization: LCSTS [10] • Question Generation:KBQG 16 , DuReader-QG [67], DuReader robust -QG [68].…”

Section: Natural Language Generation Tasksmentioning

confidence: 99%

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Sun,

Wang,

Feng

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 [1] and have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. Empirical results show that the model outperforms the state-of-the-art models on 54 Chinese NLP tasks, and its English version achieves the first place on the SuperGLUE [3] benchmark (July 3, 2021), surpassing the human performance by +0.8% (90.6% vs. 89.8%).

show abstract

“…• Text Summarization: LCSTS [10] • Question Generation:KBQG 16 , DuReader-QG [67], DuReader robust -QG [68].…”

Section: Natural Language Generation Tasksmentioning

confidence: 99%

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Sun,

Wang,

Feng

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…• Machine Reading Comprehension (MRC): DRCD [86], DuReader [87], Dureader robust [88], Dureader checklist , Dureader yesno 18 , C 3 [89], CMRC 2018 [90].…”

Section: Evaluation Tasksmentioning

confidence: 99%

ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Wang¹,

Sun²,

Xiang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Pre-trained language models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. has shown that scaling up pre-trained language models can further exploit their enormous potential. A unified framework named ERNIE 3.0 [2] was recently proposed for pre-training large-scale knowledge enhanced models and trained a model with 10 billion parameters. ERNIE 3.0 outperformed the state-of-the-art models on various NLP tasks. In order to explore the performance of scaling up ERNIE 3.0, we train a hundred-billion-parameter model called ERNIE 3.0 Titan with up to 260 billion parameters on the PaddlePaddle [3] platform. Furthermore, we design a self-supervised adversarial loss and a controllable language modeling loss to make ERNIE 3.0 Titan generate credible and controllable texts. To reduce the computation overhead and carbon emission, we propose an online distillation framework for ERNIE 3.0 Titan, where the teacher model will teach students and train itself simultaneously. ERNIE 3.0 Titan is the largest Chinese dense pre-trained model so far. Empirical results show that the ERNIE 3.0 Titan outperforms the state-of-the-art models on 68 NLP datasets.

show abstract

DuReader_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications

Cited by 2 publications

References 14 publications

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Contact Info

Product

Resources

About