Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition

Takanobu, Ryuichi; Liang, Runze; Huang, Minlie

doi:10.18653/v1/2020.acl-main.59

Cited by 34 publications

(21 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The literature on joint optimization of the DS and the US is line of research most relevant to our work. Takanobu et al (2020) proposed a hybrid value network using MARL (Lowe et al, 2017) with roleaware reward decomposition used in optimising the dialogue manager. However, their model requires separate NLU/NLG models to interact via natural language, which hinders its application in the transfer learning to new domains.…”

Section: Related Workmentioning

confidence: 99%

“…Prior work has shown benefits from this approach to dialogue policy learning, with a higher success rate at dialogue level (Liu and Lane, 2017b;Papangelis et al, 2019;Takanobu et al, 2020), but there has not been previous work that addresses multi-domain end-to-end dialogue modelling for both agents. Takanobu et al (2020) address refinement of the dialogue policy alone at the semantic level, but do not address end-to-end system architectures. Liu and Lane (2017b); Papangelis et al (2019) address single-domain dialogues (Henderson et al, 2014), but not the more realistic and complex multi-domain dialogues.…”

Section: Introductionmentioning

confidence: 99%

“…If a user model is less than perfectly optimised after supervised learning over a fixed training corpus, further learning through interaction between the two agents offers the US the opportunity to refine its behavior. Prior work has shown benefits from this approach to dialogue policy learning, with a higher success rate at dialogue level (Liu and Lane, 2017b;Papangelis et al, 2019;Takanobu et al, 2020), but there has not been previous work that addresses multi-domain end-to-end dialogue modelling for both agents. Takanobu et al (2020) address refinement of the dialogue policy alone at the semantic level, but do not address end-to-end system architectures.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Transferable Dialogue Systems and User Simulators

Tseng¹,

Dai²,

Kreyssig³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

One of the difficulties in training dialogue systems is the lack of training data. We explore the possibility of creating dialogue data through the interaction between a dialogue system and a user simulator. Our goal is to develop a modelling framework that can incorporate new dialogue scenarios through self-play between the two agents. In this framework, we first pre-train the two agents on a collection of source domain dialogues, which equips the agents to converse with each other via natural language. With further fine-tuning on a small amount of target domain data, the agents continue to interact with the aim of improving their behaviors using reinforcement learning with structured reward functions. In experiments on the MultiWOZ dataset, two practical transfer learning problems are investigated: 1) domain adaptation and 2) single-to-multiple domain transfer. We demonstrate that the proposed framework is highly effective in bootstrapping the performance of the two agents in transfer learning. We also show that our method leads to improvements in dialogue system performance on complete datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Transferable Dialogue Systems and User Simulators

Tseng¹,

Dai²,

Kreyssig³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…The dialogue literature widely applies reinforcement learning, including the recent ones based on deep architectures (Takanobu et al, 2019(Takanobu et al, , 2020Takanobu et al, 2020;Gordon-Hall et al, 2020a,b). But these taskoriented RL dialogue systems often model the dialogue with limited parameters and assumptions specific to the dataset, targeted for that task.…”

Section: Related Workmentioning

confidence: 99%

WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue

Khandelwal¹

2021

Proceedings of the 1st Workshop on Document-Grounded Dialogue and Conversational Question Answering (DialDoc 2021)

View full text Add to dashboard Cite

An intelligent dialogue system in a multi-turn setting should not only generate the responses which are of good quality, but it should also generate the responses which can lead to long-term success of the dialogue. Although, the current approaches improved the response quality, but they over-look the training signals present in the dialogue data. We can leverage these signals to generate the weakly supervised training data for learning dialog policy and reward estimator, and make the policy take actions (generates responses) which can foresee the future direction for a successful (rewarding) conversation. We simulate the dialogue between an agent and a user (modelled similar to an agent with supervised learning objective) to interact with each other. The agent uses dynamic blocking to generate ranked diverse responses and explorationexploitation to select among the Top-K responses. Each simulated state-action pair is evaluated (works as a weak annotation) with three quality modules: Semantic Relevant, Semantic Coherence and Consistent Flow. Empirical studies with two benchmarks indicate that our model can significantly out-perform the response quality and lead to a successful conversation on both automatic evaluation and human judgment. 1

show abstract

“…Over the last few years, two promising research directions in ToDs have emerged. The first focuses on a pipeline approach, which consists of modularly connected components (Wu et al, 2019a;Takanobu et al, 2020;Peng et al, 2020;. The second direction employs an end- to-end model, which directly takes the sequence-tosequence (Seq2Seq) model to generate a response from a dialogue history and a corresponding knowledge base (KB) (Eric et al, 2017;Madotto et al, 2018;Wen et al, 2018;Qin et al, 2019b;Wu et al, 2019b;Qin et al, 2020b) In recent years, with the burst of deep neural networks and the evolution of pre-trained language models, the research of ToDs has obtained great success.…”

Section: Introductionmentioning

confidence: 99%

Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Qin¹,

Xie²,

Huang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Consistency Identification has obtained remarkable success on open-domain dialogue, which can be used for preventing inconsistent response generation. However, in contrast to the rapid development in open-domain dialogue, few efforts have been made to the task-oriented dialogue direction. In this paper, we argue that consistency problem is more urgent in task-oriented domain. To facilitate the research, we introduce CI-ToD, a novel dataset for Consistency Identification in Taskoriented Dialog system. In addition, we not only annotate the single label to enable the model to judge whether the system response is contradictory, but also provide more finegrained labels (i.e., Dialogue History Inconsistency, User Query Inconsistency and Knowledge Base Inconsistency) to encourage model to know what inconsistent sources lead to it. Empirical results show that state-of-the-art methods only achieve 51.3%, which is far behind the human performance of 93.2%, indicating that there is ample room for improving consistency identification ability. Finally, we conduct exhaustive experiments and qualitative analysis to comprehend key challenges and provide guidance for future directions. All datasets and models are publicly available at https://github.com/yizhen20133868/CI-ToD.

show abstract

Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition

Cited by 34 publications

References 30 publications

Transferable Dialogue Systems and User Simulators

Transferable Dialogue Systems and User Simulators

WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue

Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

Contact Info

Product

Resources

About