Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation

Parisi, Simone; Pirotta, Matteo; Restelli, Marcello

doi:10.1613/jair.4961

Cited by 38 publications

(19 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multi-policy approaches seek to find a set of policies that cover the Pareto front. Various techniques exist, for instance repeatedly calling a single-policy approach with strategically-chosen trade-off settings [41,27,55], simultaneously learning a set of policies by using a multi-objective variant of Q-learning [26,39,54], learning a manifold of policies in parameter space [32,33], or combining single-policy approaches with an overarching objective [53].…”

Section: Related Workmentioning

confidence: 99%

On Multi-objective Policy Optimization as a Tool for Reinforcement Learning

Abdolmaleki¹,

Huang²,

Vezzani³

et al. 2021

Preprint

View full text Add to dashboard Cite

Many advances that have improved the robustness and efficiency of deep reinforcement learning (RL) algorithms can, in one way or another, be understood as introducing additional objectives, or constraints, in the policy optimization step. This includes ideas as far ranging as exploration bonuses, entropy regularization, and regularization toward teachers or data priors when learning from experts or in offline RL. Often, task reward and auxiliary objectives are in conflict with each other and it is therefore natural to treat these examples as instances of multiobjective (MO) optimization problems. We study the principles underlying MORL and introduce a new algorithm, Distillation of a Mixture of Experts (DiME), that is intuitive and scale-invariant under some conditions. We highlight its strengths on standard MO benchmark problems and consider case studies in which we recast offline RL and learning from experts as MO problems. This leads to a natural algorithmic formulation that sheds light on the connection between existing approaches. For offline RL, we use the MO perspective to derive a simple algorithm, that optimizes for the standard RL objective plus a behavioral cloning term. This outperforms state-of-the-art on two established offline RL benchmarks. * equal contribution Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

On Multi-objective Policy Optimization as a Tool for Reinforcement Learning

Abdolmaleki¹,

Huang²,

Vezzani³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In current research of multi-objective optimization (MOO) problems and multi-objective reinforcement learning (MORL), the standard concept of optimality is replaced by Pareto optimality [8,18,20]. In [18], Parisi et al formulated an RL strategy gradient method to learn Pareto boundary in multi-objective Markov decision problems, where the continuous approximation of Pareto boundary is generated for each gradient climb operation. In [20], Ruiz et al discussed a MORL method that uses non-convex Pareto boundaries to generates deterministic non-dominated strategies in multi-objective Markov decision problems.…”

Section: Related Workmentioning

confidence: 99%

Dynamic Bicycle Dispatching of Dockless Public Bicycle-sharing Systems Using Multi-objective Reinforcement Learning

Chen

et al. 2021

ACM Trans. Cyber-Phys. Syst.

View full text Add to dashboard Cite

As a new generation of Public Bicycle-sharing Systems (PBS), the Dockless PBS (DL-PBS) is an important application of cyber-physical systems and intelligent transportation. How to use artificial intelligence to provide efficient bicycle dispatching solutions based on dynamic bicycle rental demand is an essential issue for DL-PBS. In this article, we propose MORL-BD, a dynamic bicycle dispatching algorithm based on multi-objective reinforcement learning to provide the optimal bicycle dispatching solution for DL-PBS. We model the DL-PBS system from the perspective of cyber-physical systems and use deep learning to predict the layout of bicycle parking spots and the dynamic demand of bicycle dispatching. We define the multi-route bicycle dispatching problem as a multi-objective optimization problem by considering the optimization objectives of dispatching costs, dispatch truck's initial load, workload balance among the trucks, and the dynamic balance of bicycle supply and demand. On this basis, the collaborative multi-route bicycle dispatching problem among multiple dispatch trucks is modeled as a multi-agent and multi-objective reinforcement learning model. All dispatch paths between parking spots are defined as state spaces, and the reciprocal of dispatching costs is defined as a reward. Each dispatch truck is equipped with an agent to learn the optimal dispatch path in the dynamic DL-PBS network. We create an elite list to store the Pareto optimal solutions of bicycle dispatch paths found in each action, and finally get the Pareto frontier. Experimental results on the actual DL-PBS show that compared with existing methods, MORL-BD can find a higher quality Pareto frontier with less execution time.

show abstract

“…Two new manifold-based algorithms that combine episodic exploration and importance sampling were proposed to efficiently learn a manifold in the policy parameter space such that its image in the objective space accurately approximates the Pareto frontier (Parisi, Pirotta, & Peters, 2017). There are also numerous other gradient-based methods to solve multi-objective optimization (Pirotta, Parisi, & Restelli, 2015;Parisi, Pirotta, & Restelli, 2016;Parisi, et al, 2014aParisi, et al, , 2014bPinder, 2016). For example, policy gradient techniques were developed to approximate the Pareto frontier in multi-objective Markov decision processes (Pirotta et al, 2015;Parisi et al, 2016Parisi et al, , 2014aParisi et al, , 2014b.…”

Section: Previous Workmentioning

confidence: 99%

“…There are also numerous other gradient-based methods to solve multi-objective optimization (Pirotta, Parisi, & Restelli, 2015;Parisi, Pirotta, & Restelli, 2016;Parisi, et al, 2014aParisi, et al, , 2014bPinder, 2016). For example, policy gradient techniques were developed to approximate the Pareto frontier in multi-objective Markov decision processes (Pirotta et al, 2015;Parisi et al, 2016Parisi et al, , 2014aParisi et al, , 2014b. Note that an explicit quantitative study of the interobjective relationship is also lacking in these studies.…”

Section: Previous Workmentioning

confidence: 99%

Efficient Multi-objective Reinforcement Learning via Multiple-gradient Descent with Iteratively Discovered Weight-Vector Sets

Cao

Zhan²

2021

jair

View full text Add to dashboard Cite

Solving multi-objective optimization problems is important in various applications where users are interested in obtaining optimal policies subject to multiple (yet often conflicting) objectives. A typical approach to obtain the optimal policies is to first construct a loss function based on the scalarization of individual objectives and then derive optimal policies that minimize the scalarized loss function. Albeit simple and efficient, the typical approach provides no insights/mechanisms on the optimization of multiple objectives due to the lack of ability to quantify the inter-objective relationship. To address the issue, we propose to develop a new efficient gradient-based multi-objective reinforcement learning approach that seeks to iteratively uncover the quantitative inter-objective relationship via finding a minimum-norm point in the convex hull of the set of multiple policy gradients when the impact of one objective on others is unknown a priori. In particular, we first propose a new PAOLS algorithm that integrates pruning and approximate optimistic linear support algorithm to efficiently discover the weight-vector sets of multiple gradients that quantify the inter-objective relationship. Then we construct an actor and a multi-objective critic that can co-learn the policy and the multi-objective vector value function. Finally, the weight discovery process and the policy and vector value function learning process can be iteratively executed to yield stable weight-vector sets and policies. To validate the effectiveness of the proposed approach, we present a quantitative evaluation of the approach based on three case studies.

show abstract

Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation

Cited by 38 publications

References 49 publications

On Multi-objective Policy Optimization as a Tool for Reinforcement Learning

On Multi-objective Policy Optimization as a Tool for Reinforcement Learning

Dynamic Bicycle Dispatching of Dockless Public Bicycle-sharing Systems Using Multi-objective Reinforcement Learning

Efficient Multi-objective Reinforcement Learning via Multiple-gradient Descent with Iteratively Discovered Weight-Vector Sets

Contact Info

Product

Resources

About