Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues

Abstract

Robust state tracking for task-oriented dialogue systems currently remains restricted to a few popular languages. This paper shows that given a large-scale dialogue data set in one language, we can automatically produce an effective semantic parser for other languages using machine translation. We propose automatic translation of dialogue datasets with alignment to ensure faithful translation of slot values and eliminate costly human supervision used in previous benchmarks. We also propose a new contextual semantic parsing model, which encodes the formal slots and values, and only the last agent and user utterances. We show that the succinct representation reduces the compounding effect of translation errors, without harming the accuracy in practice.We evaluate our approach on several dialogue state tracking benchmarks. On RiSAWOZ, CrossWOZ, CrossWOZ-EN, and MultiWOZ-ZH datasets we improve the state of the art by 11%, 17%, 20%, and 0.3% in joint goal accuracy. We present a comprehensive error analysis for all three datasets showing erroneous annotations can obscure judgments on the quality of the model. Finally, we present RiSAWOZ English and German datasets, created using our translation methodology. On these datasets, accuracy is within 11% of the original showing that high-accuracy multilingual dialogue datasets are possible without relying on expensive human annotations.

X-RiSAWOZ: High-Quality End-to-End Multilingual Dialogue Datasets and Few-shot Agents

“…Since the dataset for target languages is introduced in this paper, there is only prior work on the Chinese dataset. In Section 7.3, we compare our results to the best previously reported result on RiSAWOZ from Moradshahi et al (2021) that achieved SOTA on the DST subtask using an mBART model, and from Quan et al (2020) for other subtasks which use DAMD , a Seq2Seq RNN end-to-end dialogue model. We use seven widely-used automatic metrics to compare different models.…”

“…A portion of their evaluation and test sets were post-edited by humans, while the training set remained entirely machine translated. Moradshahi et al (2021) translated RiSAWOZ to English and German using open-source machine translation models with alignment. However, the validation and test data were not verified by humans, resulting in potentially over-estimating the accuracy of agents.…”

“…Hybrid Alignment for NMT. Previous work (Moradshahi et al, 2021;Li et al, 2021a) proposed using alignment for tracking the position of entities during translation to ensure they can be replaced with the desired translation both in the utterance and the belief state. For this purpose, the encoder-decoder cross-attention weights of the neural machine translation model were used in a method called neural alignment.…”

Bootstrapping Multilingual Semantic Parsers using Large Language Models

“…Target logical form y i tgt has the same signature as y i eng and only differs in terms of the translated slot values. Most translation based approaches (Moradshahi et al, 2020(Moradshahi et al, , 2021Xia and Monti, 2021;Nicosia et al, 2021) translate an English example (x i eng , y i eng ) to the corresponding target language example (x i tgt , y i tgt ) via a two step process: (i) Translate: Use a supervised translation service to convert the English utterance x i eng into the target language utterance x i tgt ; and (ii) Project: Replace the English slot values in y i eng with spans copied from the translated utterance x i tgt via a learned alignment model. The translated examples are then used to train a downstream multilingual semantic parser.…”

“…A common approach to automatic multilingual dataset creation is translating existing English datasets into target languages. Prior methods utilize an off-the-shelf machine translation model for translating the English utterance into the target language x eng → x tgt , followed by projecting language specific components in the English logicalform y eng to obtain the logical-form y tgt in the target language (Moradshahi et al, 2020(Moradshahi et al, , 2021Xia and Monti, 2021;Nicosia et al, 2021;Gritta et al, 2022;. The projection step is often learned independent of the translation service, resulting in poor generalization across languages.…”