Multi-objective multiagent credit assignment in reinforcement learning and NSGA-II

Yliniemi, Logan; Tumer, Kagan

doi:10.1007/s00500-016-2124-z

Cited by 15 publications

(23 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some recent works have sought to derive coverage sets in MOMMDPs using reinforcement learning or evolutionary algorithms (e.g. [129,128,63,60,59,58]). As in single-objective MMDPs, learning joint policies which coordinate agents' actions to get the desired outcome(s) in MOMMDPs is a difficult problem.…”

Section: Team Reward -Team Utility (Trtu)mentioning

confidence: 99%

Multi-objective multi-agent decision making: a utility-based analysis and survey

Rădulescu

Mannion

Roijers

et al. 2019

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

The majority of multi-agent system (MAS) implementations aim to optimise agents' policies with respect to a single objective, despite the fact that many real-world problem domains are inherently multi-objective in nature. Multiobjective multi-agent systems (MOMAS) explicitly consider the possible trade-offs between conflicting objective functions. We argue that, in MOMAS, such compromises should be analysed on the basis of the utility that these compromises have for the users of a system. As is standard in multi-objective optimisation, we model the user utility using utility functions that map value or return vectors to scalar values. This approach naturally leads to two different optimisation criteria: expected scalarised returns (ESR) and scalarised expected returns (SER). We develop a new taxonomy which classifies multi-objective multi-agent decision making settings, on the basis of the reward structures, and which and how utility functions are applied. This allows us to offer a structured view of the field, to clearly delineate the current state-of-the-art in multi-objective multi-agent decision making approaches and to identify promising directions for future research. Starting from the execu-

show abstract

Section: Team Reward -Team Utility (Trtu)mentioning

confidence: 99%

Multi-objective multi-agent decision making: a utility-based analysis and survey

Rădulescu

Mannion

Roijers

et al. 2019

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

show abstract

“…Linear scalarization is commonly used in MORL literature (see e.g. Vamplew et al ., 2010; Roijers et al ., 2013; Van Moffaert et al ., 2013; Brys et al ., 2014; Mason et al ., 2016; Mannion et al ., 2016c, 2016d; Yliniemi & Tumer, 2016), and is defined in Equation (9) below: where w is the objective weight vector, w c is the weight for objective c , r + is the scalarized reward signal, r c is the component of the reward vector r for objective c , and C is the set of objectives. When using linear scalarization, altering the weights in the weight vector allows the user to express the relative importance of the objectives.…”

Section: Background and Related Workmentioning

confidence: 99%

“…In order to produce a set of Pareto optimal solutions using scalarized single-policy RL algorithms, researchers typically record the best non-dominated solutions found during a number of independent runs (see e.g. Vamplew et al ., 2010; Van Moffaert et al ., 2013; Yliniemi & Tumer, 2016; Mannion et al ., 2017). These solutions are then compared with one another to produce an approximation of the Pareto front.…”

Section: Background and Related Workmentioning

confidence: 99%

“…None of the works discussed thus far considered the effect of credit assignment in MOMAS. (Yliniemi & Tumer, 2016) present the first work that considers the design of reward structures in a MOMARL setting. Their work compared the effectiveness of D with that of the typical MARL reward structures L and G .…”

Section: Background and Related Workmentioning

confidence: 99%

“…The Pareto optimal or non-dominated set (NDS) consists of solutions that are incomparable, where each solution in the set is not dominated by any of the others on every objective. Examples of multi-objective problems where RL may be applied include water resource management (Mason et al ., 2016), traffic signal control (Khamis & Gomaa, 2014; Mannion et al ., 2016 a), electricity generator scheduling (Mannion et al ., 2017), supply chain management (Duggan, 2008) and robot coordination tasks (Yliniemi & Tumer, 2016).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Reward shaping for knowledge-based multi-objective multi-agent reinforcement learning

Mannion

Devlin

Duggan

et al. 2018

The Knowledge Engineering Review

View full text Add to dashboard Cite

The majority of multi-agent reinforcement learning (MARL) implementations aim to optimize systems with respect to a single objective, despite the fact that many real-world problems are inherently multi-objective in nature. Research into multi-objective MARL is still in its infancy, and few studies to date have dealt with the issue of credit assignment. Reward shaping has been proposed as a means to address the credit assignment problem in single-objective MARL, however it has been shown to alter the intended goals of a domain if misused, leading to unintended behaviour. Two popular shaping methods are potential-based reward shaping and difference rewards, and both have been repeatedly shown to improve learning speed and the quality of joint policies learned by agents in single-objective MARL domains. This work discusses the theoretical implications of applying these shaping approaches to cooperative multi-objective MARL problems, and evaluates their efficacy using two benchmark domains. Our results constitute the first empirical evidence that agents using these shaping methodologies can sample true Pareto optimal solutions in cooperative multi-objective stochastic games.

show abstract

Tuning of reinforcement learning parameters applied to SOP using the Scott–Knott method

et al. 2019

View full text Add to dashboard Cite

In this paper, we present a technique to tune the reinforcement learning (RL) parameters applied to the sequential ordering problem (SOP) using the Scott-Knott method. The RL has been widely recognized as a powerful tool for combinatorial optimization problems, such as travelling salesman and multidimensional knapsack problems. It seems, however, that less attention has been paid to solve the SOP. Here, we have developed a RL structure to solve the SOP that can partially fill that gap. Two traditional RL algorithms, Q-learning and SARSA, have been employed. Three learning specifications have been adopted to analyze the performance of the RL: algorithm type, reinforcement learning function, and parameter. A complete factorial experiment and the Scott-Knott method are used to find the best combination of factor levels, when the source of variation is statistically different in analysis of variance. The performance of the proposed RL has been tested using benchmarks from the TSPLIB library. In general, the selected parameters indicate that SARSA overwhelms the performance of Q-learning.

show abstract

Multi-objective multiagent credit assignment in reinforcement learning and NSGA-II

Cited by 15 publications

References 48 publications

Multi-objective multi-agent decision making: a utility-based analysis and survey

Multi-objective multi-agent decision making: a utility-based analysis and survey

Reward shaping for knowledge-based multi-objective multi-agent reinforcement learning

Tuning of reinforcement learning parameters applied to SOP using the Scott–Knott method

Contact Info

Product

Resources

About