Robust Dialogue State Tracking with Weak Supervision and Sparse Data

Abstract

Generalizing dialogue state tracking (DST) to new data is especially challenging due to the strong reliance on abundant and fine-grained supervision during training. Sample sparsity, distributional shift, and the occurrence of new concepts and topics frequently lead to severe performance degradation during inference. In this paper we propose a training strategy to build extractive DST models without the need for fine-grained manual span labels. Two novel input-level dropout methods mitigate the negative impact of sample sparsity. We propose a new model architecture with a unified encoder that supports value as well as slot independence by leveraging the attention mechanism. We combine the strengths of triple copy strategy DST and value matching to benefit from complementary predictions without violating the principle of ontology independence. Our experiments demonstrate that an extractive DST model can be trained without manual span labels. Our architecture and training strategies improve robustness towards sample sparsity, new concepts, and topics, leading to state-of-the-art performance on a range of benchmarks. We further highlight our model’s ability to effectively learn from non-dialogue data.

Introduction: sustainable Logistics Systems using AI-based Meta-Heuristics Approaches

“…As shown in Figure 1, experimental results on the MultiWOZ benchmark (Budzianowski et al, 2018) represent a significant milestone. Our approach is the first that, without further fine-tuning, enables modestly sized open-source LLMs (7B or 13B parameters) to achieve comparable or superior performance compared to previous state-of-the-art (SOTA) prompting methods that relied exclusively on advanced proprietary LLMs such as ChatGPT and Codex (Hudeček and Dušek, 2023;Heck et al, 2023;. Furthermore, our approach beats the previous zero-shot SOTA by 5.6% Av.…”

“…( 2) Previous prompting approaches that have only shown efficacy with advanced ChatGPT and Codex. These include IC-DST (Hu et al, 2022) using Codex, (Heck et al, 2023) and InstructTODS using ChatGPT (GPT-3.5/4).…”

“…Previous prompting approaches (Heck et al, 2023; Peng et al, 2020;Su et al, 2021) necessitate training on curated domain-specific annotated data, a process that is notoriously costly and laborintensive. Despite efforts in automated dataset creation using GPT-3 (Li et al, 2022), these methods struggle to generalize to unseen domains.…”

“…LLMs exhibit remarkable capabilities for tackling various tasks without the need for task-specific fine-tuning, making them suited for zero-shot DST. However, while there have been initiatives to leverage ChatGPT for zero-shot DST (Hu et al, 2022;Hudeček and Dušek, 2023;Heck et al, 2023;, these methods tend to treat DST as a standalone task rather than chat completion, which the models, especially chat-tuned models, are more proficient in. They usually take the whole conversation as input along with detailed instructions to generate in domain-specific formats.…”

“…TOD differs from general conversation in that it requires models to not only generate responses but also track dialogue states according to domain-specific schemas. While ChatGPT has shown effectiveness in response generation within TOD (Li et al, 2023c), their performance of zero-shot DST, as explored in recent research on prompting approaches (Hu et al, 2022;Bang et al, 2023;Hudeček and Dušek, 2023;Heck et al, 2023;, are still not satisfying, which remains a significant challenge.…”