Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems

Abstract

Dialogue policy optimisation via reinforcement learning requires a large number of training interactions, which makes learning with real users time consuming and expensive. Many set-ups therefore rely on a user simulator instead of humans. These user simulators have their own problems. While hand-coded, rule-based user simulators have been shown to be sufficient in small, simple domains, for complex domains the number of rules quickly becomes intractable. State-of-the-art datadriven user simulators, on the other hand, are still domain-dependent. This means that adaptation to each new domain requires redesigning and retraining. In this work, we propose a domain-independent transformer-based user simulator (TUS). The structure of our TUS is not tied to a specific domain, enabling domain generalisation and learning of cross-domain user behaviour from data. We compare TUS with the state of the art using automatic as well as human evaluations. TUS can compete with rule-based user simulators on pre-defined domains and is able to generalise to unseen domains in a zero-shot fashion.

GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers

“…A domain-independent transformer-based user simulator (TUS) is proposed by Lin et al (2021). With domain-independent input and output feature representations, TUS can adapt to an unseen domain in a zero-shot fashion.…”

“…The downsides of delexicalisation already became clear in early neural network dialogue state trackers (Mrkšić et al, 2017) and are further exacerbated in natural language generation (Peng et al, 2020). We do however include a rule-based user simulator (Schatzmann et al, 2007) with a template-based NLG, noted as ABUS-T in our experiments, as the rule-based user simulator has achieved competitive results in human evaluations (Kreyssig et al, 2018;Lin et al, 2021). Also, TUS (Lin et al, 2021) did not significantly outperform ABUS in the human trial, so we exclude it from the evaluation here.…”

“…We do however include a rule-based user simulator (Schatzmann et al, 2007) with a template-based NLG, noted as ABUS-T in our experiments, as the rule-based user simulator has achieved competitive results in human evaluations (Kreyssig et al, 2018;Lin et al, 2021). Also, TUS (Lin et al, 2021) did not significantly outperform ABUS in the human trial, so we exclude it from the evaluation here.…”

“…On the other hand, data-driven USs can be built with less expert involvement. However, these models are either ontology-dependent (El Asri et al, 2016;Gür et al, 2018;Kreyssig et al, 2018), which means adapting to a new domain requires reengineering the feature representation or re-training the model, or they do not model the language of the user (Lin et al, 2021). Both shortcomings are serious.…”