Scaling up deep reinforcement learning for multi-domain dialogue systems

Abstract

Abstract-Standard deep reinforcement learning methods such as Deep Q-Networks (DQN) for multiple tasks (domains) face scalability problems due to large search spaces. This paper proposes a three-stage method for multi-domain dialogue policy learning-termed NDQN, and applies it to an informationseeking spoken dialogue system in the domains of restaurants and hotels. In this method, the first stage does multi-policy learning via a network of DQN agents; the second makes use of compact state representations by compressing raw inputs; and the third stage applies a pre-training phase for bootstraping the behaviour of agents in the network. Experimental results comparing DQN (baseline) versus NDQN (proposed) using simulations report that the proposed method exhibits better scalability and is promising for optimising the behaviour of multi-domain dialogue systems. An additional evaluation reports that the NDQN agents outperformed a K-Nearest Neighbour baseline in task success and dialogue length, yielding more efficient and successful dialogues.

Deep Reinforcement Learning of Dialogue Policies with Less Weight Updates

“…word present or absent), the words derived from user responses can be seen as continuous variables by taking ASR confidence scores into account. Our state representations used delexicalised word-based representations and excluded words from information presentation-for increased scalability, as described in [20].…”

“…Firstly, actions are selected from the most likely actions, P r(a|s) > 0.0001, derived from Naive Bayes classifiers (due to scalability purposes) trained from demonstration dialogues. See example demonstration dialogue in Appendix of [20]. Secondly, the most likely actions in he previous stage are extended with legitimate requests, apologies and confirmations.…”

“…Previous work tackling faster learning of DRL-based agents have used distributed neural nets [16,17], prioritised experience replay by sampling from important previous experiences [18], fast reward propagation [19], and distributed policies to train specialised agents (one per task or domain) [20,21,22]. We propose a method that can be used on top of previous methods, which applies a reduced amount of weight updates in two ways.…”

“…This paper treats neural-based dialogue agents in multiple domains as a network of Deep Reinforcement Learners as proposed in [20], for example by using a network of Deep Qnetworks (DQN). A DQN agent aims to find an optimal policy by maximising its cumulative discounted reward defined as…”

“…While user responses can motivate transitions to another domain in the network, completing a subdialogue within a domain motivates a transition to the previous domain to resume the interaction. [20] and [22] describe algorithms to train and execute NDQN agents.…”