[CASPI] Causal-aware Safe Policy Improvement for Task-oriented Dialogue

Abstract

The recent success of reinforcement learning (RL) in solving complex tasks is often attributed to its capacity to explore and exploit an environment. Sample efficiency is usually not an issue for tasks with cheap simulators to sample data online. On the other hand, Taskoriented Dialogues (ToD) are usually learnt from offline data collected using human demonstrations. Collecting diverse demonstrations and annotating them is expensive. Unfortunately, RL policy trained on off-policy data are prone to issues of bias and generalization, which are further exacerbated by stochasticity in human response and non-markovian nature of annotated belief state of a dialogue management system. To this end, we propose a batch-RL framework for ToD policy learning: Causal-aware Safe Policy Improvement (CASPI). CASPI includes a mechanism to learn fine-grained reward that captures intention behind human response and also offers guarantee on dialogue policy's performance against a baseline. We demonstrate the effectiveness of this framework on end-to-end dialogue task of the Multiwoz2.0 dataset. The proposed method outperforms the current state of the art. Further more we demonstrate sample efficiency, where our method trained only on 20% of the data, are comparable to current state of the art method trained on 100% data on two out of there evaluation metrics.

KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learning

“…As a result, KRLS does not perform exploration/planning on a dialog-level, which can be very useful for tasks that require long-horizon planning to be successful (e.g., persuading a person to donate to a charity ). We believe one way to extend KRLS could be using a GPT-like model to learn from an entire dialog, and combine with safe policy improvement methods to avoid potentially large bias and poor sample efficiency during dialog-level RL learning (Ramachandran et al, 2022). We leave this for future work.…”

“…For appropriateness and fluency, we followed the definitions from prior work(Zhang et al, 2020b;Ramachandran et al, 2021;Jang et al, 2022;Feng et al, 2023).…”

Dialogue Evaluation with Offline Reinforcement Learning

“…The method is however only demonstrated on a very small state and action space, and it is therefore unclear whether it generalizes to more complex set ups. Ramachandran et al (2021) applied offline RL with a pair-wise reward learning model based on preference learning, however it still utilizes the corpus-based success rate for choosing the preferred rollout. To the best of our knowledge, offline RL has not previously been deployed for dialogue evaluation.…”