Priority-Objective Reinforcement Learning

Abstract

Intelligent agents often have to cope with situations in which their various needs must be prioritised. Efforts have been made, in the fields of cognitive robotics and machine learning, to model need prioritization. Examples of existing frameworks include normative decision theory, the subsumption architecture and reinforcement learning. Reinforcement learning algorithms oriented towards active goal prioritization include the options framework from hierarchical reinforcement learning and the ranking approach as well as the MORE framework from multi-objective reinforcement learning. Previous approaches can be configured to make an agent function optimally in individual environments, but cannot effectively model dynamic and efficient goal selection behaviour in a generalisable framework. Here, we propose an altered version of the MORE framework that includes a threshold constant in order to guide the agent towards making economic decisions in a broad range of 'priority-objective reinforcement learning' (PORL) scenarios. The results of our experiments indicate that pre-existing frameworks such as the standard linear scalarization, the ranking approach and the options framework are unable to induce opportunistic objective optimisation in a diverse set of environments. In particular, they display strong dependency on the exact choice of reward values at design time. However, the modified MORE framework appears to deliver adequate performance in all cases tested. From the results of this study, we conclude that employing MORE along with integrated thresholds, can effectively simulate opportunistic objective prioritization in a wide variety of contexts.

Getting Priorities Right: Intrinsic Motivation with Multi-Objective Reinforcement Learning

“…the total value for a given state is determined on the basis of a weighted sum of calculated values V k for each objective where the weighting Fig. 1: Utility concepts for the multi-objective problem: (i) standard linear scalarization, (ii) the non-linear MORE scalarization that corresponds to a softmin function [28], (iii) C-MORE, a MORE scalarization that is shifted by c 0 = 2 on the first objective, giving it higher priority [29], but also continuously weighting in the second objective when the first one becomes satisfied.…”

“…This paper investigates how intrinsic motivation can be integrated into an agent with the non-linear multi-objective reinforcement learning framework C-MORE [29]. We capitalize on C-MORE's ability to dynamically balance the achievement of different needs, while allowing for a gradual prioritisation at the same time.…”

“…The Conditional Multi-Objective Reward Exponentials (C-MORE) framework [29] makes use of a weighted exponential function for calculating the overall value of a given state in accordance with a specified priority vector. Internally, this leads to dynamic and prioritised weighting of objectives (which can be derived analytically, see [28] for details) that ensures that priority is given to the momentarily least satisfied objectives.…”

“…Non-linear methods that prevent trade-offs have however been proposed: The "Multi-Objective Reward Exponentials" (MORE) framework [28] uses a soft-min-like utility function to combine objectives, which results in a balanced achievement of all objectives. This method has also been shown to allow for targeted prioritisation [29] of objectives by the designer where appropriate, which results in behavior that satisfies urgent needs first, but giving in to secondary needs when possible.…”