Multi-objective safe reinforcement learning: the relationship between multi-objective reinforcement learning and safe reinforcement learning

Horie, Naoto; Matsui, Tetsuya; Moriyama, Koichi; Mutoh, Atsuko; Inuzuka, Nobuhiro

doi:10.1007/s10015-019-00523-3

Cited by 10 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notably, overestimation bias is a widespread phenomenon and occurs in the valuebased methods and may be effectively addressed by utilizing multiple critics in the actor-critic framework [13]. In return, this significantly limits the algorithms' scalability through the increase in the number of gradient-based updates [14]. Furthermore, the efficient Reinforcement Learning methods' essential memory complexity tends to increase linearly based on the expressive power of approximators [15] significantly.…”

Section: Model-based Rl Approachesmentioning

confidence: 99%

“…On the flip side, model-based Reinforcement Learning approaches further learn a simplified model of the surrounding world [14]. The world model allows the agent to effectively predict the outcomes of various potential action sequences, which lets it play through the other hypothetical scenarios to come up with informed decisions in new situations, resulting in the reduction of the trial and error necessary for goal achievement [17].…”

Section: Model-based Rl Approachesmentioning

confidence: 99%

“…The agent in question learns a world model from various images and utilizes it in learning longsighted behaviors [19]. Dreamer leverages its world model to ensure an efficient learning process equipping it with the necessary behaviors via backpropagation through the various model predictions [14]. Besides, it efficiently learns the required actor-network to ensure the successful forecast actions by propagating gradients of rewards by effective predicted state sequences, which is complex and showcases the scalability of RL, as shown in (Figure 1) below.…”

Section: Model-based Rl Approachesmentioning

confidence: 99%

“…Therefore, the learner can acquire up to one million queries within a second on one machine. Hence, the learner can easily be scaled to thousands of cores, such as up to 2048 on Cloud TPUs [14]. Moreover, the number of actors can also be scaled effectively to the tune of thousands of machines that fully utilize the learner and makes it possible for training to be carried out at millions of frames within every second [20].…”

Section: Performance Of Seed Rlmentioning

confidence: 99%

See 3 more Smart Citations

A Survey of Scalable Reinforcement Learning

Stone¹,

Talbert²,

Eberle³

2022

IJICR

View full text Add to dashboard Cite

Reinforcement Learning (RL) has been around for some time now, and various issues have been associated with the provision. In the last two years, RL has realized a boost in popularity through deep learning. For instance, RL played a critical role in the DeepMind AlphaGo program that was crucial to beating a top-level Go player in 2016 [1]. However, there is still work to be done despite RL's advancements before the DeepMind AlphaGo program becomes mainstream. For instance, one of the challenges of the program is its ability to multitask. [1] indicate that agents should perform various functions to achieve general AI. However, in the contemporary understanding, multitasking is one of the challenges of AI and RL scalability; for instance, it should not take over 1000 hrs. of different tasks to learn 1000 various tasks. Instead, AI agents must build up a library of general knowledge and learn general skills that are common and applicable across a variety of tasks. However, this ability is currently non-existent in such programs as the Deep Q-Network (DQN). While the DQN has previously been shown to have the ability to play various games, including Atari games, there is often no learning across the tasks. Each of the games is learned from scratch, which is often not scalable.

show abstract

Section: Model-based Rl Approachesmentioning

confidence: 99%

Section: Model-based Rl Approachesmentioning

confidence: 99%

Section: Model-based Rl Approachesmentioning

confidence: 99%

Section: Performance Of Seed Rlmentioning

confidence: 99%

See 2 more Smart Citations

A Survey of Scalable Reinforcement Learning

Stone¹,

Talbert²,

Eberle³

2022

IJICR

View full text Add to dashboard Cite

show abstract

“…For example in the DST we might prefer a policy which alternates between the (-5, 3) and (-14, 50) returns, even though the SER for this approach is lower than for the policy which mixes the (-1,1) and (-19, 124) returns. While there is prior work on reducing variance within risk-aware single-objective RL [6,21] and also on MORL approaches to risk-aware RL [35,9], we are not aware of any previous work that addresses the issue of reducing the variance in returns within the context of MORL.…”

Section: Introductionmentioning

confidence: 99%

The impact of environmental stochasticity on value-based multiobjective reinforcement learning

Vamplew

Foale

Dazeley

2021

Neural Comput & Applic

View full text Add to dashboard Cite

A common approach to address multiobjective problems using reinforcement learning methods is to extend model-free, value-based algorithms such as Q-learning to use a vector of Q-values in combination with an appropriate action selection mechanism that is often based on scalarisation. Most prior empirical evaluation of these approaches has focused on deterministic environments. This study examines the impact on stochasticity in rewards and state transitions on the behaviour of multi-objective Q-learning. It shows that the nature of the optimal solution depends on these environmental characteristics, and also on whether we desire to maximise the Expected Scalarised Return (ESR) or the Scalarised Expected Return (SER). We also identify a novel aim which may arise in some applications of maximising SER subject to satisfying constraints on the variation in return, and show that this may require different solutions than ESR or conventional SER.The analysis of the interaction between environmental stochasticity and multiobjective Q-learning is supported by empirical evaluations on several simple multiobjective Markov Decision Processes with varying characteristics. This includes a demonstration of a novel approach to learning deterministic SER-optimal policies for environments with stochastic rewards. In addition, we report a previously unidentified issue with model-free, value-based approaches to multiobjective reinforcement learning in the context of environments with stochastic state transitions. Having highlighted the limitations of value-based model-free MORL methods, we discuss several alternative methods that may be more suitable for maximising SER in MOMDPs with stochastic transitions.

show abstract

A practical guide to multi-objective reinforcement learning and planning

Hayes

Rădulescu

Bargiacchi

et al. 2022

Auton Agent Multi-Agent Syst

148

View full text Add to dashboard Cite

Real-world sequential decision-making tasks are generally complex, requiring trade-offs between multiple, often conflicting, objectives. Despite this, the majority of research in reinforcement learning and decision-theoretic planning either assumes only a single objective, or that multiple objectives can be adequately handled via a simple linear combination. Such approaches may oversimplify the underlying problem and hence produce suboptimal results. This paper serves as a guide to the application of multi-objective methods to difficult problems, and is aimed at researchers who are already familiar with single-objective reinforcement learning and planning methods who wish to adopt a multi-objective perspective on their research, as well as practitioners who encounter multi-objective decision problems in practice. It identifies the factors that may influence the nature of the desired solution, and illustrates by example how these influence the design of multi-objective decision-making systems for complex problems.

show abstract

Multi-objective safe reinforcement learning: the relationship between multi-objective reinforcement learning and safe reinforcement learning

Cited by 10 publications

References 14 publications

A Survey of Scalable Reinforcement Learning

A Survey of Scalable Reinforcement Learning

The impact of environmental stochasticity on value-based multiobjective reinforcement learning

A practical guide to multi-objective reinforcement learning and planning

Contact Info

Product

Resources

About