A framework to co-optimize task and social dialogue policies using Reinforcement Learning

Abstract

One of the main challenges for conversational agents is to select the optimal dialogue policy based on the state of the interaction. This challenge becomes even harder when the conversational agent not only has to achieve a specific task, but also aims at building rapport. Although some work already tried to tackle this challenge using a Reinforcement Learning (RL) approach, they tend to consider one single optimal policy for all the users, regardless of their conversational goals. In this work, we describe a framework that allows us to build a RL-based agent able to adapt its dialogue policy depending on its user's conversational goals. After we build a rulebased agent and a user simulator communicating at the dialog-act level, we crowdsource the surface sentences authoring for both the simulated users and the agent, which allow us to generate a dataset of interactions in natural language. Then, we annotate each of these interactions with a single rapport score and analyze the links between simulated users' conversational goals, agent conversational policies, and rapport. Our results show that rapport was higher when both or none of the interlocutors tried to build rapport. We use this result to inform the design of a social reward function, and we rely on this social reward function to train a RLbased agent using an hybrid approach of supervised learning and reinforcement learning. We evaluate our approach by comparing two different versions of our RL-based agent: one that takes users' conversational goals into account and another that does not. The results show that an agent adapting its dialogue policy depending on users' conversational goals performs better.

Designing Persuasive Food Conversational Recommender Systems With Nudging and Socially-Aware Conversational Strategies

“…Researchers have already started to investigate rapport-building conversational systems in different contexts, and study how to combine rapport-building and task-oriented strategies during an interaction ( Pecune et al, 2013 ; Zhao et al, 2018b ; Pecune and Marsella, 2020 ). With Rea the virtual Real Estate Agent, researchers investigated how small-talk influenced the price users were ready to invest in a new house ( Cassell and Bickmore, 2003 ).…”

Adapting conversational strategies in information-giving human-agent interaction

“…To optimize the dialog policy of our SIA, we rely on Reinforcement Learning. As in Pecune and Marsella (2020), Shi andYu (2018), andYu et al (2017), we build a user simulator to approximate human users' behaviors and generate enough synthetic interactions to train the agent. The use of a simulated user allows to pretrain the model and avoid the "cold start problem" where the model behaves badly in the first interactions with human users.…”

“…As shown by Pecune and Marsella (2020), people's conversational goals and preferences influence the perceived quality of an interaction. For instance, people only focusing on the task might quickly get disengaged if their interlocutor uses social conversational strategies.…”

“…None of these systems adapt their conversational behavior depending on whether their users care about the social aspect of the interaction or not, which means that the single policy learned might actually be sub-optimal. Closer to our work, the conversational recommendation agent in Pecune and Marsella ( 2020 ) learns two different dialog policies depending on its users conversational goals. Each turn, the agent selects a combination of task-oriented and socially-oriented dialog act to optimize both the task performance and the level of rapport occurring during the interaction.…”

Adapting conversational strategies to co-optimize agent's task performance and user's engagement

“…To simulate users interacting with an information-giving conversational agent, we built the following user simulator. Conversational preferences We rely on [8] to model SU's conversational preferences. Each SU has one of the following preferences: Action selection Each turn, the SU outputs a dialog act and a list of non-verbal behaviors.…”