Evaluation of a hierarchical reinforcement learning spoken dialogue system

Cuayáhuitl, Heriberto; Renals, Steve; Lemon, Oliver; Shimodaira, Hiroshi

doi:10.1016/j.csl.2009.07.001

Cited by 61 publications

(62 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There has been some prior research on using machine learning techniques in human-robot interaction, though not for multi-user interactions. One such approach involved modelling the interaction as a Semi-Markov Decision Process (SMDP) and using Hierarchical Reinforcement Learning (HRL) for optimising decision-making [Cuayáhuitl and Dethlefs 2011;Cuayáhuitl et al 2010]. An alternative approach to using multiple policies within a dialogue manager has also incorporated POMDP models, but still focused on single-user interactions [Lison 2011].…”

Section: Discussionmentioning

confidence: 99%

Machine Learning for Social Multiparty Human--Robot Interaction

Keizer

Foster

Wang

et al. 2014

ACM Trans. Interact. Intell. Syst.

Self Cite

View full text Add to dashboard Cite

We describe a variety of machine learning techniques that are being applied to social multi-user human-robot interaction, using a robot bartender in our scenario. We first present a data-driven approach to social state recognition based on supervised learning. We then describe an approach to social skills execution-i.e., action selection for generating socially appropriate robot behaviour-which is based on reinforcement learning, using a data-driven simulation of multiple users to train execution policies for social skills. Next, we describe how these components for social state recognition and skills execution have been integrated into an end-to-end robot bartender system, and we discuss the results of a user evaluation. Finally, we present an alternative unsupervised learning framework that combines social state recognition and social skills execution, based on hierarchical Dirichlet processes and an infinite POMDP interaction manager. The models make use of data from both human-human interactions collected in a number of German bars and human-robot interactions recorded in the evaluation of an initial version of the system.

show abstract

Section: Discussionmentioning

confidence: 99%

Machine Learning for Social Multiparty Human--Robot Interaction

Keizer

Foster

Wang

et al. 2014

ACM Trans. Interact. Intell. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…[15] used rule-based classifiers for predicting user intentions, which are executed using a Hierarchical Task Network incorporating expert knowledge. Trainable multi-domain dialogue systems using traditional reinforcement learning include [16], [17], [18], [19]. These systems use a modest amount of features, and in contrast to neural-based systems, they require manual feature engineering.…”

Section: Related Work and Discussionmentioning

confidence: 99%

Scaling up deep reinforcement learning for multi-domain dialogue systems

Cuayáhuitl

Williamson

et al. 2017

2017 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Abstract-Standard deep reinforcement learning methods such as Deep Q-Networks (DQN) for multiple tasks (domains) face scalability problems due to large search spaces. This paper proposes a three-stage method for multi-domain dialogue policy learning-termed NDQN, and applies it to an informationseeking spoken dialogue system in the domains of restaurants and hotels. In this method, the first stage does multi-policy learning via a network of DQN agents; the second makes use of compact state representations by compressing raw inputs; and the third stage applies a pre-training phase for bootstraping the behaviour of agents in the network. Experimental results comparing DQN (baseline) versus NDQN (proposed) using simulations report that the proposed method exhibits better scalability and is promising for optimising the behaviour of multi-domain dialogue systems. An additional evaluation reports that the NDQN agents outperformed a K-Nearest Neighbour baseline in task success and dialogue length, yielding more efficient and successful dialogues.

show abstract

“…It remains to be demonstrated how far one can go with such an approach. Future work includes to (a) compare different model architectures, training parameters and reward functions; (b) extend or improve the abilities of the proposed dialogue system; (c) train deep learning agents in other (larger scale) domains [7,8,9]; (d) evaluate end-to-end systems with real users; (e) compare or combine different types of neural nets [10]; and (e) perform fast learning based on parallel computing. Table 1 Example dialogue using the policy from Fig.2, where states are numerical representations of the last system and noisy user inputs, actions are dialogue acts, and user resposes are in brackets…”

Section: Discussionmentioning

confidence: 99%

SimpleDS: A Simple Deep Reinforcement Learning Dialogue System

Cuayáhuitl

2016

Lecture Notes in Electrical Engineering

Self Cite

View full text Add to dashboard Cite

This paper presents SimpleDS, a simple and publicly available dialogue system trained with deep reinforcement learning. In contrast to previous reinforcement learning dialogue systems, this system avoids manual feature engineering by performing action selection directly from raw text of the last system and (noisy) user responses. Our initial results, in the restaurant domain, report that it is indeed possible to induce reasonable behaviours with such an approach that aims for higher levels of automation in dialogue control for intelligent interactive agents.

show abstract

Evaluation of a hierarchical reinforcement learning spoken dialogue system

Cited by 61 publications

References 46 publications

Machine Learning for Social Multiparty Human--Robot Interaction

Machine Learning for Social Multiparty Human--Robot Interaction

Scaling up deep reinforcement learning for multi-domain dialogue systems

SimpleDS: A Simple Deep Reinforcement Learning Dialogue System

Contact Info

Product

Resources

About