Subgoal Discovery for Hierarchical Dialogue Policy Learning

Tang, Da; Li, Xiujun; Gao, Jianfeng; Wang, Chong; Li, Lihong; Jebara, Tony

doi:10.18653/v1/d18-1253

Cited by 34 publications

(22 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[23] proposed a HRL approach based on options framework to learn policies in different domains. In [24], authors propose a divide and conquer approach for efficient policy learning where a complex goal-oriented task is broken into simpler subgoals in an unsupervised manner and then these subgoals are used to learn a multi-level policy using HRL. Feudal Reinforcement Learning has been used with DQN in the work of [25] for learning policies in large domains.…”

Section: Plos Onementioning

confidence: 99%

Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning

Saha

Bhattacharyya

2020

PLoS ONE

View full text Add to dashboard Cite

Developing a Dialogue/Virtual Agent (VA) that can handle complex tasks (need) of the user pertaining to multiple intents of a domain is challenging as it requires the agent to simultaneously deal with multiple subtasks. However, majority of these end-to-end dialogue systems incorporate only user semantics as inputs in the learning process and ignore other useful user behavior and information. Sentiment of the user at the time of conversation plays an important role in securing maximum user gratification. So, incorporating sentiment of the user during the policy learning becomes even more crucial, more so when serving composite tasks of the user. Methodology As a first step towards enabling the development of sentiment aided VA for multi-intent conversations, this paper proposes a new dataset, annotated with its corresponding intents, slot and sentiment (considering the entire dialogue history) labels, named SentiVA, collected from open-sourced dialogue datasets. In order to integrate these multiple aspects, a Hierarchical Reinforcement Learning (HRL) specifically options based VA is proposed to learn strategies for managing multi-intent conversations. Along with task success based immediate rewards, sentiment based immediate rewards are also incorporated in the hierarchical value functions to make the VA user adaptive. Findings Empirically, the paper shows that task based and sentiment based immediate rewards cumulatively are required to ensure successful task completion and attain maximum user satisfaction in a multi-intent scenario instead of any of these rewards alone. Practical implications The eventual evaluators and consumers of dialogue systems are users. Thus, to ensure a fulfilling conversational experience involving maximum user satisfaction requires VA to consider user sentiment at every time-step in its decision making policy.

show abstract

Section: Plos Onementioning

confidence: 99%

Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning

Saha

Bhattacharyya

2020

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…For example, Kulkarni et al [42] presented a hierarchical-DQN framework that integrates the temporal abstraction and an intrinsic motivation mechanism to be able to play the classic Atari game ‘Montezuma’s Revenge’ which is a big challenge for standard RL approaches. Tang et al [43] used HRL to learn the dialogue policy for task-completion dialogue agents, where subgoals in the complex goal-directed task are automatically learned in an unsupervised fashion. Peng et al [44] showed that the use of subgoals mitigates the reward sparsity and leads to more effective exploration for learning.…”

Section: Related Workmentioning

confidence: 99%

Vision-Based Robot Navigation through Combining Unsupervised Learning and Hierarchical Reinforcement Learning

Zhou

Bai

Gao

et al. 2019

Sensors

View full text Add to dashboard Cite

Extensive studies have shown that many animals’ capability of forming spatial representations for self-localization, path planning, and navigation relies on the functionalities of place and head-direction (HD) cells in the hippocampus. Although there are numerous hippocampal modeling approaches, only a few span the wide functionalities ranging from processing raw sensory signals to planning and action generation. This paper presents a vision-based navigation system that involves generating place and HD cells through learning from visual images, building topological maps based on learned cell representations and performing navigation using hierarchical reinforcement learning. First, place and HD cells are trained from sequences of visual stimuli in an unsupervised learning fashion. A modified Slow Feature Analysis (SFA) algorithm is proposed to learn different cell types in an intentional way by restricting their learning to separate phases of the spatial exploration. Then, to extract the encoded metric information from these unsupervised learning representations, a self-organized learning algorithm is adopted to learn over the emerged cell activities and to generate topological maps that reveal the topology of the environment and information about a robot’s head direction, respectively. This enables the robot to perform self-localization and orientation detection based on the generated maps. Finally, goal-directed navigation is performed using reinforcement learning in continuous state spaces which are represented by the population activities of place cells. In particular, considering that the topological map provides a natural hierarchical representation of the environment, hierarchical reinforcement learning (HRL) is used to exploit this hierarchy to accelerate learning. The HRL works on different spatial scales, where a high-level policy learns to select subgoals and a low-level policy learns over primitive actions to specialize on the selected subgoals. Experimental results demonstrate that our system is able to navigate a robot to the desired position effectively, and the HRL shows a much better learning performance than the standard RL in solving our navigation tasks.

show abstract

“…Da Tang proposed Sub-goal Discovery Network [14] to divide the complex goal-oriented tasks into sub-goals in an unsupervised way. Moreover, they present a dialogue agent for the composite task of travel planning.…”

Section: Related Workmentioning

confidence: 99%

Cas-GANs: An Approach of Dialogue Policy Learning based on GAN and RL Techniques

Nabeel¹,

Riaz²,

Wang³

2019

IJACSA

View full text Add to dashboard Cite

Dialogue management systems are commonly applied in daily life, such as online shopping, hotel booking, and driving booking. Efficient dialogue management policy helps systems to respond to the user in an effective way. Policy learning is a complex task to build a dialogue system. There are different approaches have been proposed in the last decade to build a goal-oriented dialogue agent to train the systems with an efficient policy. The Generative adversarial network (GAN) is used in the dialogue generation, in previous works to build dialogue agents by selecting the optimal policy learning. The efficient dialogue policy learning aims to improve the quality of fluency and diversity for generated dialogues. Reinforcement learning (RL) algorithms are used to optimize the policies because the sequence is discrete. In this study, we have proposed a new technique called Cascade Generative Adversarial Network (Cas-GAN) that is combination of the GAN and RL for dialog generation. The Cas-GAN can model the relations between the dialogues (sentences) by using Graph Convolutional Networks (GCN). The graph nodes are consisting of different high level and low-level nodes representing the vertices and edges of the graph. Then, we use the maximum log-likelihood (MLL) approach to train the parameters and choose the best nodes. The experimental results compared with the HRL, RL agents and we got state-of-the-art results.

show abstract

Subgoal Discovery for Hierarchical Dialogue Policy Learning

Cited by 34 publications

References 36 publications

Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning

Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning

Vision-Based Robot Navigation through Combining Unsupervised Learning and Hierarchical Reinforcement Learning

Cas-GANs: An Approach of Dialogue Policy Learning based on GAN and RL Techniques

Contact Info

Product

Resources

About