Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Zhang, Shiyue; Bansal, Mohit

doi:10.18653/v1/d19-1253

Cited by 115 publications

(151 citation statements)

References 58 publications

Supporting

Mentioning

150

Contrasting

Order By: Relevance

“…• NQG-Knowledge [16], DLPH [12]: auxiliary-informationenhanced question generation models with extra inputs such as knowledge or difficulty. • Self-training-EE [38], BERT-QG-QAP [51], NQG-LM [55], CGC-QG [27] and QType-Predict [56]: multi-task question generation models with auxiliary tasks such as question answering, language modeling, question type prediction and so on.…”

Section: Evaluating Acs-aware Question Generationmentioning

confidence: 99%

See 1 more Smart Citation

Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus

Liu

Wei

Niu

et al. 2020

Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

The ability to ask questions is important in both human and machine intelligence. Learning to ask questions helps knowledge acquisition, improves question-answering and machine reading comprehension tasks, and helps a chatbot to keep the conversation flowing with a human. Existing question generation models are ineffective at generating a large amount of high-quality question-answer pairs from unstructured text, since given an answer and an input passage, question generation is inherently a one-to-many mapping. In this paper, we propose Answer-Clue-Style-aware Question Generation (ACS-QG), which aims at automatically generating high-quality and diverse question-answer pairs from unlabeled text corpus at scale by imitating the way a human asks questions. Our system consists of: i) an information extractor, which samples from the text multiple types of assistive information to guide question generation; ii) neural question generators, which generate diverse and controllable questions, leveraging the extracted assistive information; and iii) a neural quality controller, which removes low-quality generated data based on text entailment. We compare our question generation models with existing approaches and resort to voluntary human evaluation to assess the quality of the generated question-answer pairs. The evaluation results suggest that our system dramatically outperforms state-of-the-art neural question generation models in terms of the generation quality, while being scalable in the meantime. With models trained on a relatively smaller amount of data, we can generate 2.8 million quality-assured question-answer pairs from a million sentences found in Wikipedia.

show abstract

Section: Evaluating Acs-aware Question Generationmentioning

confidence: 99%

“…[55] incorporates language modeling task to help question generation. [51] utilizes question paraphrasing and question answering tasks to regularize the QG model to generate semantically valid questions.…”

Section: Related Workmentioning

confidence: 99%

Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus

Liu

Wei

Niu

et al. 2020

Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

show abstract

“…Rewards. We use ROUGE-L, QPP, and QAP (Zhang and Bansal, 2019) as rewards for this task. QPP is calculated as the probability of the generated question being the paraphrase of the ground-truth question via a classifier trained on Quora Question Pairs.…”

Section: Question Generationmentioning

confidence: 99%

“…For pre-processing, we do standard tokenization. We report on evaluation metrics including BLEU-4, METEOR, ROUGE-L, Q-BLEU1 (Nema and Khapra, 2018), as well as QPP and QAP (Zhang and Bansal, 2019).…”

Section: Question Generationmentioning

confidence: 99%

DORB: Dynamically Optimizing Multiple Rewards with Bandits

Pasunuru

Guo

Bansal

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

Policy gradients-based reinforcement learning has proven to be a promising approach for directly optimizing non-differentiable evaluation metrics for language generation tasks. However, optimizing for a specific metric reward leads to improvements in mostly that metric only, suggesting that the model is gaming the formulation of that metric in a particular way without often achieving real qualitative improvements. Hence, it is more beneficial to make the model optimize multiple diverse metric rewards jointly. While appealing, this is challenging because one needs to manually decide the importance and scaling weights of these metric rewards. Further, it is important to consider using a dynamic combination and curriculum of metric rewards that flexibly changes over time. Considering the above aspects, in our work, we automate the optimization of multiple metric rewards simultaneously via a multi-armed bandit approach (DORB), where at each round, the bandit chooses which metric reward to optimize next, based on expected arm gains. We use the Exp3 algorithm for bandits and formulate two approaches for bandit rewards: (1) Single Multi-reward Bandit (SM-Bandit); (2) Hierarchical Multi-reward Bandit (HM-Bandit). We empirically show the effectiveness of our approaches via various automatic metrics and human evaluation on two important NLG tasks: question generation and data-to-text generation, including on an unseen-test transfer setup. Finally, we present interpretable analyses of the learned bandit curriculum over the optimized rewards.

show abstract

“…Synthetic data generation has also been used for question answering (QA) tasks [2,19,43,47]. Much effort has focused on the machine reading comprehension (MRC) variant of QA, where questions should be answered in the context of a prose paragraph.…”

Section: Introductionmentioning

confidence: 99%

Sanitizing Synthetic Training Data Generation for Question Answering over Knowledge Graphs

Linjordet

Balog

2020

Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

Synthetic data generation is important to training and evaluating neural models for question answering over knowledge graphs. The quality of the data and the partitioning of the datasets into training, validation and test splits impact the performance of the models trained on this data. If the synthetic data generation depends on templates, as is the predominant approach for this task, there may be a leakage of information via a shared basis of templates across data splits if the partitioning is not performed hygienically. This paper investigates the extent of such information leakage across data splits, and the ability of trained models to generalize to test data when the leakage is controlled. We find that information leakage indeed occurs and that it affects performance. At the same time, the trained models do generalize to test data under the sanitized partitioning presented here. Importantly, these findings extend beyond the particular flavor of question answering task we studied and raise a series of difficult questions around template-based synthetic data generation that will necessitate additional research.

show abstract

Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Cited by 115 publications

References 58 publications

Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus

Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus

DORB: Dynamically Optimizing Multiple Rewards with Bandits

Sanitizing Synthetic Training Data Generation for Question Answering over Knowledge Graphs

Contact Info

Product

Resources

About