A Practical Guide to Multi-Objective Reinforcement Learning and Planning

Abstract

Real-world decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multiobjective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems.

Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)

“…The limitations of scalar rewards and the advantages of vector-based multi-objective rewards for computational agents have been extensively established in prior work [39,56,58]. In the interests of space and brevity, we focus here on the aspects that are of most relevance to the reward-is-enough hypothesis.…”

“…The prevalence of genuinely multi-objective tasks in the real-world is reflected in the thriving nature of research fields such as multi-criteria decision-making [16,70,75] and multi-objective optimisation [18,25]. Furthermore, this is reflected in the broad range of application areas that involve multi-objective aspects, as identified in [17,39], which span almost all aspects of human society. In particular, any decision which affects multiple stake-holders will require consideration of multiple objectives [19].…”

“…In particular, some forms of utility such as lexicographic ordering cannot be represented as a scalar value [26]. In contrast, intelligence based on vector rewards and approaches that are explicitly multi-objective can directly optimise any desired measure of utility [39].…”

“…Being able to explicitly identify the trade-off underlying an agent's basis for reasoning has long been argued as a key component of transparency in AGI [41]. The review and adjust scheme [39] has three major phases. During the planning or learning phase, an AGI would utilise a multi-objective reinforcement learning or planning algorithm to compute a set of optimal policies for all possible utility functions.…”

Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey

“…This may be a single goal or a hierarchy of goals, each of which can be cycled through to generate the explanation of its behaviour. A goal may be aligned to a single objective or to multiple objective that must be balanced [101]. The agent switch between these goals/objectives due to internal changes in priorities or progression in solving a larger goal.…”