The Need for MORE: Need Systems as Non-Linear Multi-Objective Reinforcement Learning

Reimann, Matthias

doi:10.1109/icdl-epirob48136.2020.9278062

Cited by 4 publications

(28 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fig. 2: Utility concepts for the multi-objective problem: (i) standard linear scalarization [8], (ii) ranking, in which the first objective always dominates the second one until a threshold of c 0 = 2 is reached, (iii) the non-linear MORE scalarization that corresponds to a softmin function [8], (iv) a MORE scalarization that is shifted by c 0 = 2 on the first objective, giving it higher priority, but also continuously weighting in the second objective when the first one becomes satisfied.…”

Section: Methodsmentioning

confidence: 99%

“…For example the ranking approach [6] makes use of thresholds and objective ordering to encourage the agent to optimize objectives one at a time. The "Multi-Objective Reward Exponentials" (MORE) framework [8], on the other hand, takes a more dynamic and continuous approach whereby the accumulated past rewards for all the objectives are actively used for dynamically weighting the future values of a given objective. The values of each objective are transformed according to an exponential function which acts as a deficit model, and makes the agent focus on the least achieved objective.…”

Section: Multi-objective Reinforcement Learningmentioning

confidence: 99%

“…The values of each objective are transformed according to an exponential function which acts as a deficit model, and makes the agent focus on the least achieved objective. The optimal weight dynamics to ensure a balanced need achievement can be derived analytically in this case [8].…”

Section: Multi-objective Reinforcement Learningmentioning

confidence: 99%

“…This paper investigates how the non-linearity of MORE [8] can be utilized to achieve a robust objective prioritization that is not possible with linear approaches. In particular, we demonstrate that this can be achieved by an operation as simple as shifting the reward related to one objective up or down by a threshold constant.…”

Section: Contribution and Outlinementioning

confidence: 99%

“…We use the term 'Priority-Objective Reinforcement Learning' (PORL) to refer to all frameworks in which an agent switches objectives according to certain conditions in the environment. In this study we explore three different methods of implementing PORL namely, the ranking approach [6], the options framework [7] and a modified version of the MORE framework [8]. We also use the standard linear scalarization of objectives as a baseline.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Priority-Objective Reinforcement Learning

Al-Husaini

Reimann

2021

2021 IEEE International Conference on Development and Learning (ICDL)

Self Cite

View full text Add to dashboard Cite

Intelligent agents often have to cope with situations in which their various needs must be prioritised. Efforts have been made, in the fields of cognitive robotics and machine learning, to model need prioritization. Examples of existing frameworks include normative decision theory, the subsumption architecture and reinforcement learning. Reinforcement learning algorithms oriented towards active goal prioritization include the options framework from hierarchical reinforcement learning and the ranking approach as well as the MORE framework from multi-objective reinforcement learning. Previous approaches can be configured to make an agent function optimally in individual environments, but cannot effectively model dynamic and efficient goal selection behaviour in a generalisable framework. Here, we propose an altered version of the MORE framework that includes a threshold constant in order to guide the agent towards making economic decisions in a broad range of 'priority-objective reinforcement learning' (PORL) scenarios. The results of our experiments indicate that pre-existing frameworks such as the standard linear scalarization, the ranking approach and the options framework are unable to induce opportunistic objective optimisation in a diverse set of environments. In particular, they display strong dependency on the exact choice of reward values at design time. However, the modified MORE framework appears to deliver adequate performance in all cases tested. From the results of this study, we conclude that employing MORE along with integrated thresholds, can effectively simulate opportunistic objective prioritization in a wide variety of contexts.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Multi-objective Reinforcement Learningmentioning

confidence: 99%

Section: Multi-objective Reinforcement Learningmentioning

confidence: 99%

Section: Contribution and Outlinementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Priority-Objective Reinforcement Learning

Al-Husaini

Reimann

2021

2021 IEEE International Conference on Development and Learning (ICDL)

Self Cite

View full text Add to dashboard Cite

show abstract

An Improved Multi-objective Optimization Algorithm Based on Reinforcement Learning

Liu

Zhou

Qiu

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Multi-objective reinforcement learning (MORL) aims to find a set of high-performing and diverse policies that address trade-offs between multiple conflicting objectives. However, in practice, decision makers (DMs) often deploy only one or a limited number of trade-off policies. Providing too many diversified trade-off policies to the DM not only significantly increases their workload but also introduces noise in multi-criterion decision-making. With this in mind, we propose a humanin-the-loop policy optimization framework for preference-based MORL that interactively identifies policies of interest. Our method proactively learns the DM's implicit preference information without requiring any a priori knowledge, which is often unavailable in real-world black-box decision scenarios. The learned preference information is used to progressively guide policy optimization towards policies of interest. We evaluate our approach against three conventional MORL algorithms that do not consider preference information and four state-of-the-art preference-based MORL algorithms on two MORL environments for robot control and smart grid management. Experimental results fully demonstrate the effectiveness of our proposed method in comparison to the other peer algorithms.

show abstract

Using soft maximin for risk averse multi-objective decision-making

Smith

Klassert

Roland³

2022

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

Balancing multiple competing and conflicting objectives is an essential task for any artificial intelligence tasked with satisfying human values or preferences. Conflict arises both from misalignment between individuals with competing values, but also between conflicting value systems held by a single human. Starting with principle of loss-aversion, we designed a set of soft maximin function approaches to multi-objective decision-making. Bench-marking these functions in a set of previously-developed environments, we found that one new approach in particular, ‘split-function exp-log loss aversion’ (SFELLA), learns faster than the state of the art thresholded alignment objective method Vamplew (Engineering Applications of Artificial Intelligenceg 100:104186, 2021) on three of four tasks it was tested on, and achieved the same optimal performance after learning. SFELLA also showed relative robustness improvements against changes in objective scale, which may highlight an advantage dealing with distribution shifts in the environment dynamics. We further compared SFELLA to the multi-objective reward exponentials (MORE) approach, and found that SFELLA performs similarly to MORE in a simple previously-described foraging task, but in a modified foraging environment with a new resource that was not depleted as the agent worked, SFELLA collected more of the new resource with very little cost incurred in terms of the old resource. Overall, we found SFELLA useful for avoiding problems that sometimes occur with a thresholded approach, and more reward-responsive than MORE while retaining its conservative, loss-averse incentive structure.

show abstract

The Need for MORE: Need Systems as Non-Linear Multi-Objective Reinforcement Learning

Cited by 4 publications

References 30 publications

Priority-Objective Reinforcement Learning

Priority-Objective Reinforcement Learning

An Improved Multi-objective Optimization Algorithm Based on Reinforcement Learning

Using soft maximin for risk averse multi-objective decision-making

Contact Info

Product

Resources

About