BERT for Question Generation

Chan, Ying-Hong; Fan, Yao-Chung

doi:10.18653/v1/w19-8624

Cited by 16 publications

(9 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, applying large-scale pretrained models in QG attracts more and more researchers' interests. Chan et al build a recurrent BERT to output one question word at a recurrent step [11,12], but it is time-consuming. The generative pretrained models such as UNILM [18], T5 [57], PEGASUS [82], and UNILMV2 [5] report the model's QG scores finetuned on SQuAD [58] dataset, but they do not explore the idea of building a unified QG.…”

Section: Related Work 21 Question Generationmentioning

confidence: 99%

Unified Question Generation with Continual Lifelong Learning

Yuan,

Yin,

et al. 2022

Preprint

View full text Add to dashboard Cite

Question Generation (QG), as a challenging Natural Language Processing task, aims at generating questions based on given answers and context. Existing QG methods mainly focus on building or training models for specific QG datasets. These works are subject to two major limitations: (1) They are dedicated to specific QG formats (e.g., answer-extraction or multi-choice QG), therefore, if we want to address a new format of QG, a re-design of the QG model is required. (2) Optimal performance is only achieved on the dataset they were just trained on. As a result, we have to train and keep various QG models for different QG datasets, which is resource-intensive and ungeneralizable.To solve the problems, we propose a model named Unified-QG based on lifelong learning techniques, which can continually learn QG tasks across different datasets and formats. Specifically, we first build a format-convert encoding to transform different kinds of QG formats into a unified representation. Then, a method named STRIDER (SimilariT y RegularI zed Difficult Example Replay) is built to alleviate catastrophic forgetting in continual QG learning. Extensive experiments were conducted on 8 QG datasets across 4 QG formats (answer-extraction, answer-abstraction, multi-choice, and boolean QG) to demonstrate the effectiveness of our approach. Experimental results demonstrate that our Unified-QG can effectively and continually adapt to QG tasks when datasets and formats vary. In addition, we verify the ability of a single trained Unified-QG model in improving 8 Question Answering (QA) systems' performance through generating synthetic QA data.

show abstract

Section: Related Work 21 Question Generationmentioning

confidence: 99%

Unified Question Generation with Continual Lifelong Learning

Yuan,

Yin,

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These massively overparameterized neural networks have revolutionized many different NLP tasks. Effective application of BERT in NMT has been studied in a number of contemporary research projects; Language Modeling, Named Entity Recognition, Question Answering, Natural Language Inference, Text Classification (Devlin et al, 2019), and Question Generation (Chan and Fan, 2019). We approach this problem from the novel perspective of extracting linguistic information encoded in BERT and applying such information in NMT.…”

Section: Nmt and Bertmentioning

confidence: 99%

Better Neural Machine Translation by Extracting Linguistic Information from BERT

Shavarani¹,

Sarkar²

2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

View full text Add to dashboard Cite

Adding linguistic information (syntax or semantics) to neural machine translation (NMT) has mostly focused on using point estimates from pre-trained models. Directly using the capacity of massive pre-trained contextual word embedding models such as BERT (Devlin et al., 2019) has been marginally useful in NMT because effective fine-tuning is difficult to obtain for NMT without making training brittle and unreliable. We augment NMT by extracting dense fine-tuned vector-based linguistic information from BERT instead of using point estimates. Experimental results show that our method of incorporating linguistic information helps NMT to generalize better in a variety of training contexts and is no more difficult to train than conventional Transformerbased NMT.

show abstract

Section: Nmt and Bertmentioning

confidence: 99%

Better Neural Machine Translation by Extracting Linguistic Information from BERT

Shavarani¹,

Sarkar²

2021

Preprint

View full text Add to dashboard Cite

Adding linguistic information (syntax or semantics) to neural machine translation (NMT) has mostly focused on using point estimates from pre-trained models. Directly using the capacity of massive pre-trained contextual word embedding models such as BERT (Devlin et al., 2019) has been marginally useful in NMT because effective fine-tuning is difficult to obtain for NMT without making training brittle and unreliable. We augment NMT by extracting dense finetuned vector-based linguistic information from BERT instead of using point estimates. Experimental results show that our method of incorporating linguistic information helps NMT to generalize better in a variety of training contexts and is no more difficult to train than conventional Transformer-based NMT.

show abstract

BERT for Question Generation

Cited by 16 publications

References 5 publications

Unified Question Generation with Continual Lifelong Learning

Unified Question Generation with Continual Lifelong Learning

Better Neural Machine Translation by Extracting Linguistic Information from BERT

Better Neural Machine Translation by Extracting Linguistic Information from BERT

Contact Info

Product

Resources

About