Robustness Testing of Language Understanding in Task-Oriented Dialog

Abstract

Most language understanding models in dialog systems are trained on a small amount of annotated training data, and evaluated in a small set from the same distribution. However, these models can lead to system failure or undesirable outputs when being exposed to natural perturbation in practice. In this paper, we conduct comprehensive evaluation and analysis with respect to the robustness of natural language understanding models, and introduce three important aspects related to language understanding in real-world dialog systems, namely, language variety, speech characteristics, and noise perturbation. We propose a model-agnostic toolkit LAUG to approximate natural perturbation for testing the robustness issues in dialog systems. Four data augmentation approaches covering the three aspects are assembled in LAUG, which reveals critical robustness issues in state-of-theart models. The augmented dataset through LAUG can be used to facilitate future research on the robustness testing of language understanding in dialog systems.

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

“…Schema-guided modeling aims to build task-oriented dialogue systems that can generalize easily to new verticals using very little extra information, including for slot filling (Bapna et al 2017;Shah et al 2019;Liu et al 2020) and dialogue state tracking (Li et al 2021;Campagna et al 2020;Kumar et al 2020) among other tasks. More recent work has adopted the schema-guided paradigm (Ma et al 2019;Li, Xiong, and Cao 2020;Zhang et al 2021) and even extended the paradigm in functionality (Mosig, Mehri, and Kober 2020;Mehri and Eskenazi 2021).…”

“…As they are inherently public-facing in nature, the robustness of dialogue systems to harmful inputs (Dinan et al 2019;Cheng, Wei, and Hsieh 2019) and input noise (Einolghozati et al 2019;Liu et al 2020), such as ASR error, misspellings, and user input paraphrasing have been explored. However, robustness to API schemas for schema-guided dialogue systems remains relatively unexplored.…”

Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering

“…Contrastive pre-training identifies and eliminates biased tokens at the input representation level. However, this does not change the entity label distribution, which is another source of bias that makes deep learning models brittle to unseen scenarios [25,33]. To address this, we adapt the adversarial filtering proposed by [39] to smooth the entity label distribution to prevent the model from learning only from the head of the distribution (frequent entities) but also from the tail of the distribution (rarer entities).…”

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

“…As they are inherently public-facing in nature, dialogue system robustness has been explored along harmful inputs (Dinan et al 2019;Cheng, Wei, and Hsieh 2019) and input noise (Einolghozati et al 2019;Liu et al 2020), such as ASR error, misspellings, and user input paraphrasing. API schemas, however, are different from utterances as dialogue inputs, and unique to zero/few-shot models.…”

“…Schema-guided modeling builds on work on building task-oriented dialogue systems that can generalize easily to new verticals using very little extra information, including for slot filling (Bapna et al 2017;Shah et al 2019;Liu et al 2020) and dialogue state tracking (Li et al 2021;Campagna et al 2020;Kumar et al 2020) among other tasks. More recent work has adopted the schema-guided paradigm (Ma et al 2019;Li, Xiong, and Cao 2020;Zhang et al 2021) and even extended the paradigm in functionality (Mosig, Mehri, and Kober 2020;Mehri and Eskenazi 2021).…”