The SPPD System for Schema Guided Dialogue State Tracking Challenge

Li, Miao; Xiong, Haoqi; Cao, Yunbo

doi:10.48550/arxiv.2006.09035

Cited by 3 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Schema-guided modeling aims to build task-oriented dialogue systems that can generalize easily to new verticals using very little extra information, including for slot filling (Bapna et al 2017;Shah et al 2019;Liu et al 2020) and dialogue state tracking (Li et al 2021;Campagna et al 2020;Kumar et al 2020) among other tasks. More recent work has adopted the schema-guided paradigm (Ma et al 2019;Li, Xiong, and Cao 2020;Zhang et al 2021) and even extended the paradigm in functionality (Mosig, Mehri, and Kober 2020;Mehri and Eskenazi 2021).…”

Section: Related Workmentioning

confidence: 99%

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

Lee

Gupta

Rastogi

et al. 2022

AAAI

View full text Add to dashboard Cite

Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research. The Schema-Guided Dialogue (SGD) dataset introduced a paradigm for enabling models to support any service in zero-shot through schemas, which describe service APIs to models in natural language. We explore the robustness of dialogue systems to linguistic variations in schemas by designing SGD-X - a benchmark extending SGD with semantically similar yet stylistically diverse variants for every schema. We observe that two top state tracking models fail to generalize well across schema variants, measured by joint goal accuracy and a novel metric for measuring schema sensitivity. Additionally, we present a simple model-agnostic data augmentation method to improve schema robustness.

show abstract

Section: Related Workmentioning

confidence: 99%

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

Lee

Gupta

Rastogi

et al. 2022

AAAI

View full text Add to dashboard Cite

Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research. The Schema-Guided Dialogue (SGD) dataset introduced a paradigm for enabling models to support any service in zero-shot through schemas, which describe service APIs to models in natural language. We explore the robustness of dialogue systems to linguistic variations in schemas by designing SGD-X - a benchmark extending SGD with semantically similar yet stylistically diverse variants for every schema. We observe that two top state tracking models fail to generalize well across schema variants, measured by joint goal accuracy and a novel metric for measuring schema sensitivity. Additionally, we present a simple model-agnostic data augmentation method to improve schema robustness.

show abstract

“…Schema-guided modeling builds on work on building task-oriented dialogue systems that can generalize easily to new verticals using very little extra information, including for slot filling (Bapna et al 2017;Shah et al 2019;Liu et al 2020) and dialogue state tracking (Li et al 2021;Campagna et al 2020;Kumar et al 2020) among other tasks. More recent work has adopted the schema-guided paradigm (Ma et al 2019;Li, Xiong, and Cao 2020;Zhang et al 2021) and even extended the paradigm in functionality (Mosig, Mehri, and Kober 2020;Mehri and Eskenazi 2021). Lin et al (2021) and Cao and Zhang (2021) both investigate different natural language description styles for dialogue state tracking generalization.…”

Section: Related Workmentioning

confidence: 99%

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

Lee,

Gupta,

Rastogi

et al. 2021

Preprint

View full text Add to dashboard Cite

Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research. The Schema-Guided Dialogue (SGD) dataset introduced a paradigm for enabling models to support an unlimited number of services without additional data collection or re-training through the use of schemas. Schemas describe service APIs in natural language, which models consume to understand the services they need to support. However, the impact of the choice of language in these schemas on model performance remains unexplored. We address this by releasing SGD-X, a benchmark for measuring the robustness of dialogue systems to linguistic variations in schemas. SGD-X extends the SGD dataset with crowdsourced variants for every schema, where variants are semantically similar yet stylistically diverse. We evaluate two dialogue state tracking models on SGD-X and observe that neither generalizes well across schema variations, measured by joint goal accuracy and a novel metric for measuring schema sensitivity. Furthermore, we present a simple modelagnostic data augmentation method to improve schema robustness and zero-shot generalization to unseen services.

show abstract

“…To tackle the challenge of the zero-shot slot filling, we leverage the power of the pre-trained NLP models, compute complex bi-directional relationships of utterance and slot types, and contextualize the multi-granular information to better accommodate unseen concepts. In a related, but orthogonal line of research, the authors in [15,28,34] tackled the problem of slot filling in the context of dialog state tracking where dialog state and history are available in addition to an input utterance. In contrast, this work and the SOTA models we compare against in our experiments only consider an utterance without having access to any dialog state elements.…”

Section: Related Workmentioning

confidence: 99%

Linguistically-Enriched and Context-Aware Zero-shot Slot Filling

Siddique¹,

Jamour²,

Hristidis³

2021

Preprint

View full text Add to dashboard Cite

Slot filling is identifying contiguous spans of words in an utterance that correspond to certain parameters (i.e., slots) of a user request/query. Slot filling is one of the most important challenges in modern task-oriented dialog systems. Supervised learning approaches have proven effective at tackling this challenge, but they need a significant amount of labeled training data in a given domain. However, new domains (i.e., unseen in training) may emerge after deployment. Thus, it is imperative that these models seamlessly adapt and fill slots from both seen and unseen domains -unseen domains contain unseen slot types with no training data, and even seen slots in unseen domains are typically presented in different contexts. This setting is commonly referred to as zero-shot slot filling. Little work has focused on this setting, with limited experimental evaluation. Existing models that mainly rely on contextindependent embedding-based similarity measures fail to detect slot values in unseen domains or do so only partially. We propose a new zero-shot slot filling neural model, LEONA, which works in three steps.Step one acquires domain-oblivious, context-aware representations of the utterance word by exploiting (a) linguistic features such as part-of-speech; (b) named entity recognition cues; and (c) contextual embeddings from pre-trained language models.Step two fine-tunes these rich representations and produces slotindependent tags for each word.Step three exploits generalizable context-aware utterance-slot similarity features at the word level, uses slot-independent tags, and contextualizes them to produce slot-specific predictions for each word. Our thorough evaluation on four diverse public datasets demonstrates that our approach consistently outperforms the state-of-the-art models by 17.52%, 22.15%, 17.42%, and 17.95% on average for unseen domains on SNIPS, ATIS, MultiWOZ, and SGD datasets, respectively.

show abstract

The SPPD System for Schema Guided Dialogue State Tracking Challenge

Cited by 3 publications

References 6 publications

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

Linguistically-Enriched and Context-Aware Zero-shot Slot Filling

Contact Info

Product

Resources

About