Multi-objective infinite-horizon discounted Markov decision processes

White, D. J.

doi:10.1016/0022-247x(82)90122-6

Cited by 69 publications

(80 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These formalisms have been extended [11,13,6] with parameter uncertainty. Here, we consider the formalism of Givan et al [6]; other models are more powerful.…”

Section: Related Workmentioning

confidence: 99%

The Complexity of Uncertainty in Markov Decision Processes

Scheftelowitsch¹

2015

2015 Proceedings of the Conference on Control and Its Applications

View full text Add to dashboard Cite

We consider Markov decision processes with uncertain transition probabilities and two optimization problems in this context: the finite horizon problem which asks to find an optimal policy for a finite number of transitions and the percentile optimization problem for a wide class of uncertain Markov decision processes which asks to find a policy with the optimal probability to reach a given reward objective. To the best of our knowledge, unlike other optimality criteria, the finite horizon problem has not been considered for the case of bounded-parameter Markov decision processes, and the percentile optimization problem has only been considered for very special cases. Unlike most problems in the Markov decision process research context, dynamic programming is not applicable, as the usual subdivision in independent subproblems in each state is not anymore possible. Justified by this observation, we establish NP-hardness results for these problems by showing appropriate reductions.

show abstract

“…These formalisms have been extended [11,13,6] with parameter uncertainty. Here, we consider the formalism of Givan et al [6]; other models are more powerful.…”

Section: Related Workmentioning

confidence: 99%

The Complexity of Uncertainty in Markov Decision Processes

Scheftelowitsch¹

2015

2015 Proceedings of the Conference on Control and Its Applications

View full text Add to dashboard Cite

show abstract

“…These ideas are related to earlier work on solving multi-criteria MDPs where weights are used to indicate the importance of different reward components. For example, in the work by White (1982), vector-based generalizations of successive approximation techniques are used to solved the MDP. Feinberg and Schwartz (1995) formulate the problem as optimizing a weighted sum of the total discounted rewards for the different components of the reward function.…”

Section: Related Workmentioning

confidence: 99%

Transfer in variable-reward hierarchical reinforcement learning

et al. 2008

View full text Add to dashboard Cite

Transfer learning seeks to leverage previously learned tasks to achieve faster learning in a new task. In this paper, we consider transfer learning in the context of related but distinct Reinforcement Learning (RL) problems. In particular, our RL problems are derived from Semi-Markov Decision Processes (SMDPs) that share the same transition dynamics but have different reward functions that are linear in a set of reward features. We formally define the transfer learning problem in the context of RL as learning an efficient algorithm to solve any SMDP drawn from a fixed distribution after experiencing a finite number of them. Furthermore, we introduce an online algorithm to solve this problem, Variable-Reward Reinforcement Learning (VRRL), that compactly stores the optimal value functions for several SMDPs, and uses them to optimally initialize the value function for a new SMDP. We generalize our method to a hierarchical RL setting where the different SMDPs share the same task hierarchy. Our experimental results in a simplified real-time strategy domain show that significant transfer learning occurs in both flat and hierarchical settings. Transfer is especially effective in the hierarchical setting where the overall value functions are decomposed into subtask value functions which are more widely amenable to transfer across different SMDPs.

show abstract

“…, w k , the pure memoryless optimal strategy for the single objective i w i r i is Pareto-optimal. This technique is called the weighted factor method [6,12], and used commonly in engineering practice to find subsets of the Pareto set [7]. However, not all Pareto-optimal points are obtained in this fashion, as the following example shows that randomization is necessary.…”

Section: Propositionmentioning

confidence: 99%

“…We study MDPs with multiple long-run average objectives, an extension of the MDP model where there are several reward functions [6,12]. In MDPs with multiple objectives, we are interested not in a single solution that is simultaneously optimal in all objectives (which may not exist), but in a notion of "tradeoffs" called the Pareto curve.…”

Section: Introductionmentioning

confidence: 99%

Markov Decision Processes with Multiple Long-Run Average Objectives

Chatterjee

2007

FSTTCS 2007: Foundations of Software Technology and Theoretical Computer Science

View full text Add to dashboard Cite

Abstract. We consider Markov decision processes (MDPs) with multiple long-run average objectives. Such MDPs occur in design problems where one wishes to simultaneously optimize several criteria, for example, latency and power. The possible trade-offs between the different objectives are characterized by the Pareto curve. We show that every Pareto optimal point can be ε-approximated by a memoryless strategy, for all ε > 0. In contrast to the single-objective case, the memoryless strategy may require randomization. We show that the Pareto curve can be approximated (a) in polynomial time in the size of the MDP for irreducible MDPs; and (b) in polynomial space in the size of the MDP for all MDPs. Additionally, we study the problem if a given value vector is realizable by any strategy, and show that it can be decided in polynomial time for irreducible MDPs and in NP for all MDPs. These results provide algorithms for design exploration in MDP models with multiple long-run average objectives.

show abstract

Multi-objective infinite-horizon discounted Markov decision processes

Cited by 69 publications

References 3 publications

The Complexity of Uncertainty in Markov Decision Processes

The Complexity of Uncertainty in Markov Decision Processes

Transfer in variable-reward hierarchical reinforcement learning

Markov Decision Processes with Multiple Long-Run Average Objectives

Contact Info

Product

Resources

About