MultiWOZ 2.3: A Multi-domain Task-Oriented Dialogue Dataset Enhanced with Annotation Corrections and Co-Reference Annotation

Han, Ting; Liu, Ximing; Takanobu, Ryuichi; Lian, Yixin; Huang, Chongxuan; Wan, Dazhen; Peng, Wei; Huang, Minlie

doi:10.1007/978-3-030-88483-3_16

Cited by 23 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It covers multiple domains, consists of a large amount of dialogs, and has been chosen as benchmark for many dialog tasks, e.g. dialog state tracking (Zhang et al, , 2020aHeck et al, 2020), dialog policy optimization (yang Wu et al, 2019;Wang et al, 2020a,b) and end-to-end dialog modeling (Zhang et al, 2020b;Hosseini-Asl et al, 2020;Peng et al, 2020;Huang et al, 2021). And to polish it up to be a better benchmark, many works pay effort to improve and correct dataset (Eric et al, 2020;Zang et al, 2020;Qian et al, 2021;Han et al, 2021;Ye et al, 2021).…”

Section: Impact Of Entity Addressing Methodsmentioning

confidence: 99%

Database Search Results Disambiguation for Task-Oriented Dialog Systems

Qian¹,

Beirami²,

Kottur³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

As task-oriented dialog systems are becoming increasingly popular in our lives, more realistic tasks have been proposed and explored. However, new practical challenges arise. For instance, current dialog systems cannot effectively handle multiple search results when querying a database, due to the lack of such scenarios in existing public datasets. In this paper, we propose Database Search Result (DSR) Disambiguation, a novel task that focuses on disambiguating database search results, which enhances user experience by allowing them to choose from multiple options instead of just one. To study this task, we augment the popular task-oriented dialog datasets (MultiWOZ and SGD) with turns that resolve ambiguities by (a) synthetically generating turns through a pre-defined grammar, and (b) collecting human paraphrases for a subset. We find that training on our augmented dialog data improves the model's ability to deal with ambiguous scenarios, without sacrificing performance on unmodified turns. Furthermore, pre-fine tuning and multi-task learning help our model to improve performance on DSR-disambiguation even in the absence of in-domain data, suggesting that it can be learned as a universal dialog skill. Our data and code will be made publicly available.

show abstract

Section: Impact Of Entity Addressing Methodsmentioning

confidence: 99%

Database Search Results Disambiguation for Task-Oriented Dialog Systems

Qian¹,

Beirami²,

Kottur³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…dialog state tracking Heck et al, 2020), dialog policy optimization (yang Wu et al, 2019;Wang et al, 2020a,b) and end-to-end dialog modeling (Zhang et al, 2020b;Hosseini-Asl et al, 2020;Peng et al, 2020;. And to polish it up to be a better benchmark, many works pay effort to improve and correct dataset (Eric et al, 2020;Zang et al, 2020;Qian et al, 2021;Han et al, 2021;Ye et al, 2021). In this paper, we choose MultiWOZ 2.2 version to conduct augmentation.…”

Section: Impact Of Entity Addressing Methodsmentioning

confidence: 99%

Database Search Results Disambiguation for Task-Oriented Dialog Systems

Qian¹,

Beirami²,

Kottur³

et al. 2021

Preprint

View full text Add to dashboard Cite

As task-oriented dialog systems are becoming increasingly popular in our lives, more realistic tasks have been proposed and explored. However, new practical challenges arise. For instance, current dialog systems cannot effectively handle multiple search results when querying a database, due to lack of such scenarios in existing public datasets. In this paper, we propose Database Search Result (DSR) Disambiguation, a novel task that focuses on disambiguating database search results, which enhances user experience by allowing them to choose from multiple options instead of just one. To study this task, we augment the popular task-oriented dialog datasets (Multi-WOZ and SGD) with turns that resolve ambiguities by (a) synthetically generating turns through a pre-defined grammar, and (b) collecting human paraphrases for a subset. We find that training on our augmented dialog data improves the model's ability to deal with ambiguous scenarios, without sacrificing performance on unmodified turns. Furthermore, pre-fine tuning and multi-task learning helps our model to improve performance on DSRdisambiguation even in the absence of indomain data, suggesting that it can be learned as a universal dialog skill. Our data and code will be made publicly available.

show abstract

“…Research in task-oriented dialog has been, for a long time, limited by the existence of only monolingual English datasets. While earlier datasets focused on a single domain (Henderson et al, 2014a,b;Wen et al, 2017), the focus shifted towards the more realistic multi-domain task-oriented dialogs with the creation of the Mul-tiWOZ dataset , which has been refined and improved in several iterations Zang et al, 2020;Han et al, 2021). Due to the particularly high costs of creating TOD datasets (in comparison with other language understanding tasks) (Razumovskaia et al, 2021), only a handful of monolingual TOD datasets Table 5: Per-language few-shot transfer performance (sample efficiency results) on DST and RR for the baseline TOD-XLMR and the best specialized model (TLM+RS-Mono on OS).…”

Section: Few-shot Transfer and Sample Efficiencymentioning

confidence: 99%

Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog

Hung¹,

Lauscher²,

Vulić³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Research on (multi-domain) task-oriented dialog (TOD) has predominantly focused on the English language, primarily due to the shortage of robust TOD datasets in other languages, preventing the systematic investigation of cross-lingual transfer for this crucial NLP application area. In this work, we introduce MULTI 2 WOZ, a new multilingual multidomain TOD dataset, derived from the wellestablished English dataset MULTIWOZ, that spans four typologically diverse languages: Chinese, German, Arabic, and Russian. In contrast to concurrent efforts (Ding et al., 2021;Zuo et al., 2021), MULTI 2 WOZ contains goldstandard dialogs in target languages that are directly comparable with development and test portions of the English dataset, enabling reliable and comparative estimates of cross-lingual transfer performance for TOD. We then introduce a new framework for multilingual conversational specialization of pretrained language models (PrLMs) that aims to facilitate crosslingual transfer for arbitrary downstream TOD tasks. Using such conversational PrLMs specialized for concrete target languages, we systematically benchmark a number of zero-shot and few-shot cross-lingual transfer approaches on two standard TOD tasks: Dialog State Tracking and Response Retrieval. Our experiments show that, in most setups, the best performance entails the combination of (i) conversational specialization in the target language and (ii) few-shot transfer for the concrete TOD task. Most importantly, we show that our conversational specialization in the target language allows for an exceptionally sample-efficient fewshot transfer for downstream TOD tasks.

show abstract

MultiWOZ 2.3: A Multi-domain Task-Oriented Dialogue Dataset Enhanced with Annotation Corrections and Co-Reference Annotation

Cited by 23 publications

References 24 publications

Database Search Results Disambiguation for Task-Oriented Dialog Systems

Database Search Results Disambiguation for Task-Oriented Dialog Systems

Database Search Results Disambiguation for Task-Oriented Dialog Systems

Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog

Contact Info

Product

Resources

About