SimpleDS: A Simple Deep Reinforcement Learning Dialogue System

Abstract

This paper presents SimpleDS, a simple and publicly available dialogue system trained with deep reinforcement learning. In contrast to previous reinforcement learning dialogue systems, this system avoids manual feature engineering by performing action selection directly from raw text of the last system and (noisy) user responses. Our initial results, in the restaurant domain, report that it is indeed possible to induce reasonable behaviours with such an approach that aims for higher levels of automation in dialogue control for intelligent interactive agents.

Scaling up deep reinforcement learning for multi-domain dialogue systems

“…Previous work on dialogue policy learning using DRL map raw (noisy) text to actions [2], [7]. This is not only computationally intensive, but it becomes infeasible for dialogue systems with large vocabularies.…”

“…The proposed framework for training multi-domain neuralbased dialogue agents is a substantial extension from the publicly available software tools SimpleDS [2] and ConvnetJS [11]. It can be executed in training or test mode using simulations or speech-based interactions (via a mobile App 1 ).…”

“…Rather than learning with whole action sets, our framework supports learning from constrained actions by applying learning updates only on the set of valid actions. These actions are derived from the most likely actions, P r(a|s) > 0.0001, from Naive Bayes classifiers (due to scalability purposes) 1 https://youtu.be/B5fZfZ-xaKM 2 The unique words in our system's vocabulary excludes words from information presentation due to the vast amount of information about hotels and restaurants. Nonetheless and during testing, our system retrieves live information from http://www.bookatable.co.uk) and www.reservetravel.com.…”

“…This paper makes use of raw noisy text as features in an attempt to avoid engineered features to represent the dialogue state. By using this representation, dialogue agents bypass spoken language understanding in order to learn dialogue policies directly from raw (noisy) text to actions [2].…”

“…Raw features in human-machine conversations such as words with confidence scores can be given as input to a reinforcement learning agent to induce dialogue policies from interaction with the environment, where situations (words) are mapped to actions (dialogue acts) by maximizing a long-term reward signal [2]. An RL agent is typically characterized by: (i) a finite set of states S = {s 1 , ..., s n }; (ii) a finite set of actions A = {a 1 , ..., a m }; (iii) a state transition function T (s, a, s ) that specifies the next state s given the current state s and action a; (iv) a reward function R(s, a, s ) that specifies the reward given to the agent for choosing action a when the environment makes a transition from state s to state s ; and (v) a policy π : S → A that defines a mapping from states to actions.…”