A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation

Abstract

Natural language generation lies at the core of generative dialogue systems and conversational agents. We describe an ensemble neural language generator, and present several novel methods for data representation and augmentation that yield improved results in our model. We test the model on three datasets in the restaurant, TV and laptop domains, and report both objective and subjective evaluations of our best model. Using a range of automatic metrics, as well as human evaluators, we show that our approach achieves better results than state-of-the-art models on the same datasets.

Semantic Noise Matters for Neural Natural Language Generation

“…Evaluating the Impact on Neural NLGWe chose two recent neural end-to-end NLG systems, which represent two different approaches to semantic control and have been widely used and extended by the research community.4 Note that this can be done automatically, unlike fixing the references to match the original MRs.5 Juraska et al (2018)'s script reaches 6.2% SER and 60 instances with errors, most of which is just omitting the eat-Type[restaurant] value. If we ignore this value, it gets 1.9% SER and 20 incorrect instances.…”

A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models

“…Our BLEU scores are about 15 points below the baselines on the Laptops dataset and 20 points below the baselines on the TVs dataset. Upon examing the evaluation script in detail we see that BLEU score is calculated using 5 model outputs which Juraska et al (2018) and Wen et al (2016) do. We only produce the 1-best output at test time, perhaps explaining the difference.…”

“…We show results for both greedy and beam decoding with beam size 8 under p 0 and p 1 models. We compare our models to the best sequence-to-sequence DNN model, Slug (Juraska et al, 2018), the best grammar rule based model, DANGNT (Nguyen and Tran, 2018), and the best template based model, TUDA (Puzikov and Gurevych, 2018), as determined during the shared task evaluation (Dušek et al, 2019). Surprisingly, p 0 using greedy decoding surpases all of the baseline systems.…”

“…Data augmentation has also been used by prior neural generation models. Juraska et al (2018) breaks multi-sentence utterances into separate training instances. They also try training on more complex sentences alone but this model was less reliably able to realize all attributes correctly.…”

Iterative Delexicalization for Improved Spoken Language Understanding

“…Still it does not alleviate the problem of learning OOD slots since the overall task is again trained on the training data distribution. Using delexicalization to improve performance of other NLP systems has also been studied in the context of natural language generation [Juraska et al, 2018], dependency parsing [Zeman and Resnik, 2008;Denis and Dehouck, 2017], semantic parsing [Dong and Lapata, 2016], and representation learning . In this paper, we develop a novel algorithm to iteratively delexicalize the input utterance guided by model's confidence on the current input as well as utterance context.…”