A Mixture-of-Expert Approach to RL-based Dialogue Management

Abstract

Despite recent advancements in language models (LMs), their application to dialogue management (DM) problems and ability to carry on rich conversations remain a challenge. We use reinforcement learning (RL) to develop a dialogue agent that avoids being short-sighted (outputting generic utterances) and maximizes overall user satisfaction. Most existing RL approaches to DM train the agent at the word-level, and thus, have to deal with a combinatorially complex action space even for a medium-size vocabulary. As a result, they struggle to produce a successful and engaging dialogue even if they are warm-started with a pre-trained LM. To address this issue, we develop a RL-based DM using a novel mixture of expert language model (MoE-LM) that consists of (i) a LM capable of learning diverse semantics for conversation histories, (ii) a number of specialized LMs (or experts) capable of generating utterances corresponding to a particular attribute or personality, and (iii) a RL-based DM that performs dialogue planning with the utterances generated by the experts. Our MoE approach provides greater flexibility to generate sensible utterances with different intents and allows RL to focus on conversational-level DM. We compare it with SOTA baselines on open-domain dialogues and demonstrate its effectiveness both in terms of the diversity and sensibility of the generated utterances and the overall DM performance.Preprint. Under review.

DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization

“…A diversified approach would obfuscate the agent's strategies, improving its competitive edge (Lanctot et al 2017). Furthermore, in domains like dialogue systems, monotony can dull user interactions, whereas varied responses could significantly enhance user experience (Li et al 2016;Gao et al 2019;Pavel, Budulan, and Rebedea 2020;Xu et al 2022;Chow et al 2022).…”