End-to-end optimization of goal-driven and visually grounded dialogue systems

Abstract

End-to-end design of dialogue systems has recently become a popular research topic thanks to powerful tools such as encoder-decoder architectures for sequence-to-sequence learning. Yet, most current approaches cast human-machine dialogue management as a supervised learning problem, aiming at predicting the next utterance of a participant given the full history of the dialogue. This vision may fail to correctly render the planning problem inherent to dialogue as well as its contextual and grounded nature. In this paper, we introduce a Deep Reinforcement Learning method to optimize visually grounded task-oriented dialogues, based on the policy gradient algorithm. This approach is tested on the question generation task from the dataset GuessWhat?! containing 120k dialogues and provides encouraging results at solving both the problem of generating natural dialogues and the task of discovering a specific object in a complex image.

What's to Know? Uncertainty as a Guide to Asking Goal-Oriented Questions

“…We ask three human subjects to play on the same split and the game is recognised as successful if at least two of them give the right answer. In our experiment, the average performance of humans was 79% compared to 52% and 70% for the supervised [9] and RL [26] models. We are even better than a model proposed in [37] (76%), which has three complex hand-crafted rewards.…”

“…This improvement is because the question generator has the chance to better explore possible questions. Additionally, the greedy approach outperforms others in the RL baseline in [26]. This illustrates that the distribution of the words obtained from the softmax in the question generator is not very peaked and the difference between the best and second best word is often small.…”

“…Since our approach seeks uncertain words, those words are exploited at training time, which leads to lower variance (a more peaked distribution) and better performance of the greedy selection. Beam search significantly increases performance when we carry out 5 rounds (as in [9,26]) of question-answering. This is because the most informative words are selected by our approach which, combined with the beam-search's mechanism for forward exploration, leads to better performance.…”

Learning Goal-Oriented Visual Dialog Agents: Imitating and Surpassing Analytic Experts

“…To train a questioner for solving the GuessWhat?! game, [3,7] construct an "oracle" network to mimic the answerer's behavior, regard it as part of the environment in the reinforcement learning setup and then apply the REINFORCE algorithm (or Monte Carlo Policy Gradient). The questioner learns to ask critical questions that help identify the target object by interacting with the oracle.…”

Efficient Dialog Policy Learning via Positive Memory Retention

“…This progress, in turn, generated an emerging research area, the learning of goal-oriented dialogs [6]. This research involves agents that conduct a multi-turn dialogue to achieve some task-specific goal, such as locating a specific object in a group of objects [7], inferring which image the user is thinking about [8], and providing customer services and restaurant reservations [6]. All these tasks require that the agent possesses the ability to conduct a multi-round dialog and to track the inter-dependence of each questionanswer pair.…”