Sample-efficient batch reinforcement learning for dialogue management optimization

Abstract

Spoken Dialogue Systems (SDS) are systems which have the ability to interact with human beings using natural language as the medium of interaction. A dialogue policy plays a crucial role in determining the functioning of the dialogue management module. Handcrafting the dialogue policy is not always an option, considering the complexity of the dialogue task and the stochastic behavior of users. In recent years approaches based on Reinforcement Learning (RL) for policy optimization in dialogue management have been proved to be an efficient approach for dialogue policy optimization. Yet most of the conventional RL algorithms are data intensive and demand techniques such as user simulation. Doing so, additional modeling errors are likely to occur. This paper explores the possibility of using a set of approximate dynamic programming algorithms for policy optimization in SDS. Moreover, these algorithms are combined to a method for learning a sparse representation of the value function. Experimental results show that these algorithms when applied to dialogue management optimization are particularly sample efficient , since they learn from few hundreds of dialogue examples. These algorithms learn in an off-policy manner, meaning that they can learn optimal policies with dialogue examples generated with a quite simple strategy. Thus they can learn good dialogue policies directly from data , avoiding user modeling errors.

Context‐Sensitive Natural Language Generation: From Knowledge‐Driven to Data‐Driven Techniques

“…The Scalability Challenge is again related to the current absence of powerful methods in the field that can deal with large amounts of data or knowledge in a scalable way. Several approaches exist that aim to tackle complex and ambitious goals ( Janarthanam and Lemon 2010;Rieser et al 2010;Pietquin et al 2011), but they are currently confined to smallscale applications. 5.…”

Integrated Learning of Dialog Strategies and Semantic Parsing

“…We use KTD-Q (Kalman Temporal Difference Qlearning (Geist and Pietquin, 2010)) to learn the dialog policy as it was designed to satisfy some of these properties and tested in a dialog system with simulated users (Pietquin et al, 2011). The properties we wished to be satisfied by the algorithm were the following:…”

Policy Optimization for Spoken Dialog Management Using Genetic Algorithm

“…One of the solutions proposed is to use batch RL [19] or hybrid learning [20] to learn DM policies directly on dialog (s, a) corpora. We propose a fitness function to support batchstyle policy learning using GA. First, a batch RL algorithm is performed on the corpus, inducing an estimated optimum Q-functionQ(s, a), and an corresponding implicitly defined policy π Q (s) = arg max aQ (s, a).…”

“…Fitted Q-iteration (FQI) [21] is utilized for batch RL and is described in Algorithm 2. FQI has been applied to DM policies optimization and shows high dataefficiency [19]. The inputs to the algorithm are state-actionnext-state triplets in the form of {(s i,t , a i,t , s i,t+1 )}.…”