On Multi-objective Policy Optimization as a Tool for Reinforcement Learning

Abstract

Many advances that have improved the robustness and efficiency of deep reinforcement learning (RL) algorithms can, in one way or another, be understood as introducing additional objectives, or constraints, in the policy optimization step. This includes ideas as far ranging as exploration bonuses, entropy regularization, and regularization toward teachers or data priors when learning from experts or in offline RL. Often, task reward and auxiliary objectives are in conflict with each other and it is therefore natural to treat these examples as instances of multiobjective (MO) optimization problems. We study the principles underlying MORL and introduce a new algorithm, Distillation of a Mixture of Experts (DiME), that is intuitive and scale-invariant under some conditions. We highlight its strengths on standard MO benchmark problems and consider case studies in which we recast offline RL and learning from experts as MO problems. This leads to a natural algorithmic formulation that sheds light on the connection between existing approaches. For offline RL, we use the MO perspective to derive a simple algorithm, that optimizes for the standard RL objective plus a behavioral cloning term. This outperforms state-of-the-art on two established offline RL benchmarks. * equal contribution Preprint. Under review.

Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)

“…This is particularly concerning given that Silver et al are highly influential researchers and employed at DeepMind, one of the organisations best equipped to expand the frontiers of AGI. While Silver et al "hope that other researchers will join us on our quest", we instead hope that the creation of AGI based on reward maximisation is tempered by other researchers with an understanding of the issues of AI safety [45,47] and an appreciation of the benefits of multi-objective agents [1,2].…”

How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation

“…[9], [10], [11], [12], [13], [14]. While most studies only consider learning over simulated environments and lowdimensional state features, recent studies that do show successful learning from high-dimensional observations (such as images) are actor-critic algorithms with constraints or regularization on the policy [10], [9], [12], [15]. Our methods build on some of these: the exponential advantage-weighted actor-critic formulation from CRR [10] and AWAC [9].…”

“…[21] for an overview) in which the RL objective is augmented to maximize reward while staying close to a prior policy. This prior can be instantiated with a suboptimal teacher policy, which is often used for transfer learning [22], [15].…”

“…[23], [24]). In particular, we build upon the MPO algorithm [24], a state-of-the-art off-policy algorithm that, as we show below, can be adapted for the kickstarting setting and can also be related to offline RL algorithms [15].…”

Flexible Multiple-Objective Reinforcement Learning for Chip Placement

“…In [3], DQNs are again revisited with a focus on generalization across domains. MORL is cast as a mixture of expert synthesis problems with behaviour cloning in [17].…”