Goal-Oriented Multi-Task BERT-Based Dialogue State Tracker

Gulyaev, P. Yu.; Elistratova, Eugenia; Konovalov, V. P.; Kuratov, Yuri; Pugachev, Leonid; Burtsev, Mikhail

doi:10.48550/arxiv.2002.02450

Cited by 10 publications

(12 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our experiment, we consider both retrieval and generative models, which covers most of search system application. However, it is still not comprehensive because several state-of-arts fine-grained approaches, such as a BERT-based system [24], are not considered in this paper. However, given that our selected systems cover a wide spectrum of differing performances and are of various types (i.e., retrieval and generative models), we believe that the incorporation of these new models would not bring significant changes to our meta-evaluation results.…”

Section: Discussionmentioning

confidence: 99%

Meta-evaluation of Conversational Search Evaluation Metrics

Liu

Zhou

Wilson

2021

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Conversational search systems, such as Google assistant and Microsoft Cortana, enable users to interact with search systems in multiple rounds through natural language dialogues. Evaluating such systems is very challenging, given that any natural language responses could be generated, and users commonly interact for multiple semantically coherent rounds to accomplish a search task. Although prior studies proposed many evaluation metrics, the extent of how those measures effectively capture user preference remain to be investigated. In this article, we systematically meta-evaluate a variety of conversational search metrics. We specifically study three perspectives on those metrics: (1) reliability : the ability to detect “actual” performance differences as opposed to those observed by chance; (2) fidelity : the ability to agree with ultimate user preference; and (3) intuitiveness : the ability to capture any property deemed important: adequacy, informativeness, and fluency in the context of conversational search. By conducting experiments on two test collections, we find that the performance of different metrics vary significantly across different scenarios, whereas consistent with prior studies, existing metrics only achieve weak correlation with ultimate user preference and satisfaction. METEOR is, comparatively speaking, the best existing single-turn metric considering all three perspectives. We also demonstrate that adapted session-based evaluation metrics can be used to measure multi-turn conversational search, achieving moderate concordance with user satisfaction. To our knowledge, our work establishes the most comprehensive meta-evaluation for conversational search to date.

show abstract

Section: Discussionmentioning

confidence: 99%

Meta-evaluation of Conversational Search Evaluation Metrics

Liu

Zhou

Wilson

2021

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

show abstract

“…In contrast, we leverage rich commonsense knowledge graph to capture deep semantic and discriminative relationships between utterances and intents, which significantly reduces the bias towards classifying unseen intents into seen ones. In a related, but orthogonal, line of research, the authors in (Ma et al, 2019;Gulyaev et al, 2020) addressed the problem of intent detection in the context of dialog state tracking where dialog state and conversation history are available in addition to an input utterance. In contrast, this work and the SOTA models we compare against in our experiments only consider an utterance without having access to any dialog state elements.…”

Section: Related Workmentioning

confidence: 99%

Generalized Zero-shot Intent Detection via Commonsense Knowledge

Siddique

Jamour

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Identifying user intents from natural language utterances is a crucial step in conversational systems that has been extensively studied as a supervised classification problem. However, in practice, new intents emerge after deploying an intent detection model. Thus, these models should seamlessly adapt and classify utterances with both seen and unseen intents -unseen intents emerge after deployment and they do not have training data. The few existing models that target this setting rely heavily on the training data of seen intents and consequently overfit to these intents, resulting in a bias to misclassify utterances with unseen intents into seen ones. We propose RIDE: an intent detection model that leverages commonsense knowledge in an unsupervised fashion to overcome the issue of training data scarcity. RIDE computes robust and generalizable relationship meta-features that capture deep semantic relationships between utterances and intent labels; these features are computed by considering how the concepts in an utterance are linked to those in an intent label via commonsense knowledge. Our extensive experimental analysis on three widely-used intent detection benchmarks shows that relationship meta-features significantly improve the detection of both seen and unseen intents and that RIDE outperforms the state-of-the-art models.

show abstract

“…Since the models are pre-trained on large corpora, they demonstrate strong abilities to produce good results when transferred to downstream tasks. In view of this, the research of DST has been shifted to building new models on top of these powerful pre-trained language models [15,21,25,27,29,43,48,58]. For example, SUMBT [27] employs BERT to learn the relationships between slots and dialogue utterances through a slot-word attention mechanism.…”

Section: Related Workmentioning

confidence: 99%

Slot Self-Attentive Dialogue State Tracking

Manotumruksa

Zhang

et al. 2021

Preprint

View full text Add to dashboard Cite

An indispensable component in task-oriented dialogue systems is the dialogue state tracker, which keeps track of users' intentions in the course of conversation. The typical approach towards this goal is to fill in multiple pre-defined slots that are essential to complete the task. Although various dialogue state tracking methods have been proposed in recent years, most of them predict the value of each slot separately and fail to consider the correlations among slots. In this paper, we propose a slot self-attention mechanism that can learn the slot correlations automatically. Specifically, a slottoken attention is first utilized to obtain slot-specific features from the dialogue context. Then a stacked slot self-attention is applied on these features to learn the correlations among slots. We conduct comprehensive experiments on two multi-domain task-oriented dialogue datasets, including MultiWOZ 2.0 and MultiWOZ 2.1. The experimental results demonstrate that our approach achieves stateof-the-art performance on both datasets, verifying the necessity and effectiveness of taking slot correlations into consideration. CCS CONCEPTS• Computing methodologies → Natural language processing; Discourse, dialogue and pragmatics; Machine learning.

show abstract

Goal-Oriented Multi-Task BERT-Based Dialogue State Tracker

Cited by 10 publications

References 3 publications

Meta-evaluation of Conversational Search Evaluation Metrics

Meta-evaluation of Conversational Search Evaluation Metrics

Generalized Zero-shot Intent Detection via Commonsense Knowledge

Slot Self-Attentive Dialogue State Tracking

Contact Info

Product

Resources

About