AgentGraph: Toward Universal Dialogue Management With Structured Deep Reinforcement Learning

Abstract

Dialogue policy plays an important role in taskoriented spoken dialogue systems. It determines how to respond to users. The recently proposed deep reinforcement learning (DRL) approaches have been used for policy optimization. However, these deep models are still challenging for two reasons: 1) Many DRL-based policies are not sample-efficient. 2) Most models don't have the capability of policy transfer between different domains. In this paper, we propose a universal framework, AgentGraph, to tackle these two problems. The proposed AgentGraph is the combination of GNN-based architecture and DRL-based algorithm. It can be regarded as one of the multiagent reinforcement learning approaches. Each agent corresponds to a node in a graph, which is defined according to the dialogue domain ontology. When making a decision, each agent can communicate with its neighbors on the graph. Under AgentGraph framework, we further propose Dual GNN-based dialogue policy, which implicitly decomposes the decision in each turn into a high-level global decision and a low-level local decision. Experiments show that AgentGraph models significantly outperform traditional reinforcement learning approaches on most of the 18 tasks of the PyDial benchmark. Moreover, when transferred from the source task to a target task, these models not only have acceptable initial performance but also converge much faster on the target task.

Variational Denoising Autoencoders and Least-Squares Policy Iteration for Statistical Dialogue Managers

“…However, a fair comparison can be made with selected recent works utilizing PyDial in the same environments as shown in Table II. The reported accuracies for both the Feudal-DQN combination in [15] and the AgentGraph methods such as the FM-DGNN in [25] with 4000 episodes, for the environments 1, 3 and 6, which correspond to our 0%, 15% and 30% SER, are consistently lower than the ones reported here for 3000 episodes. The performance gains of our VDAE-LSPI increase for higher noise levels and for more complex domains such as the LAP11.…”

LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations

“…HAN (Wang et al, 2019b) converts the original heterogeneous graph into multiple homogeneous graphs and applies a hierarchical attention mechanism to the meta-path-based sub-graphs. Similar ideas have been adopted in dialogue state tracking (Chen et al, 2020b(Chen et al, , 2019a, dialogue policy learning (Chen et al, 2018) and text matching (Chen et al, 2020c;Lyu et al, 2021) to handle heterogeneous inputs. In another branch, Chen et al (2019b), Zhu et al (2019) and Zhao et al (2020) construct the line graph of the original graph and explicitly model the computation over edge features.…”

Prior Knowledge Driven Label Embedding for Slot Filling in Natural Language Understanding

“…• There are a few other kinds of prior knowledge, like the hierarchical dependency of domain-intent-slot [40]. Graph neural networks (GNN) have been applied to encode structured data [62], [63]. Applying GNN to encode structured knowledge is one of the future work.…”