The impact of environmental stochasticity on value-based multiobjective reinforcement learning

Abstract

A common approach to address multiobjective problems using reinforcement learning methods is to extend model-free, value-based algorithms such as Q-learning to use a vector of Q-values in combination with an appropriate action selection mechanism that is often based on scalarisation. Most prior empirical evaluation of these approaches has focused on deterministic environments. This study examines the impact on stochasticity in rewards and state transitions on the behaviour of multi-objective Q-learning. It shows that the nature of the optimal solution depends on these environmental characteristics, and also on whether we desire to maximise the Expected Scalarised Return (ESR) or the Scalarised Expected Return (SER). We also identify a novel aim which may arise in some applications of maximising SER subject to satisfying constraints on the variation in return, and show that this may require different solutions than ESR or conventional SER.The analysis of the interaction between environmental stochasticity and multiobjective Q-learning is supported by empirical evaluations on several simple multiobjective Markov Decision Processes with varying characteristics. This includes a demonstration of a novel approach to learning deterministic SER-optimal policies for environments with stochastic rewards. In addition, we report a previously unidentified issue with model-free, value-based approaches to multiobjective reinforcement learning in the context of environments with stochastic state transitions. Having highlighted the limitations of value-based model-free MORL methods, we discuss several alternative methods that may be more suitable for maximising SER in MOMDPs with stochastic transitions.

Expected Scalarised Returns Dominance: A New Solution Concept for Multi-Objective Decision Making

“…In order to apply RL to the real world the MORL community must consider the ESR criterion. However, the ESR criterion has largely been ignored by the MORL community, with the exception of the works of Roijers et al [31,30], Hayes et al [17,16] and Vamplew et al [39]. The works of Hayes et al [16,17] and Roijers et al [30] present single-policy algorithms that are suitable to learn policies under the ESR criterion, however, prior to this work, a formal definition of the necessary requirements to satisfy the ESR criterion had not previously been defined.…”

“…The current MORL literature focuses only on methods which learn the optimal set of policies under the SER criterion [23,41]. As already highlighted, the ESR criterion has largely been ignored by the MORL community, with a few exceptions [30,17,39]. In Section 6 we address this research gap and we present a novel multi-objective tabular distributional reinforcement learning (MOTDRL) algorithm that learns an optimal set of policies for the ESR criterion, also known as the ESR set, for multi-objective multi-armed bandit (MOMAB) problems.…”

Special issue on adaptive and learning agents 2020

“…In the seventh paper, The impact of environmental stochasticity on value-based multiobjective reinforcement learning [7], Vamplew et al analyse the role of stochastic rewards and stochastic state transitions in multi-objective Q-Learning. Most of the previous empirical evaluations of multi-objective reinforcement learning and scalarisation methods assume that environments are deterministic.…”

“…Most of the previous empirical evaluations of multi-objective reinforcement learning and scalarisation methods assume that environments are deterministic. In [7], the authors find that stochasticity in rewards/transitions affects the optimal solution agents which can learn and, importantly, introduce important differences based on the choice of optimisation criterion (e.g. expected scalarised returns or scalarised expected returns).…”

A Practical Guide to Multi-Objective Reinforcement Learning and Planning

“…Nevertheless, this violates the assumption of additive returns in the Bellman equation at the heart of these algorithms [Roijers et al, 2013], and therefore it may be necessary to condition the Q-values and the agent's choice of action on an augmented state formed by concatenating the environmental state with the summed rewards previously received by the agent [Geibel, 2006]. Additionally these approaches may fail to converge to the optimal policy in environments with stochastic state transitions [Vamplew et al, 2021a].…”