Two Views on Multiple Mean-Payoff Objectives in Markov Decision Processes

Abstract

We study Markov decision processes (MDPs) with multiple limit-average (or mean-payoff) functions. We consider two different objectives, namely, expectation and satisfaction objectives. Given an MDP with k k k reward functions, in the expectation objective the goal is to maximize the expected limit-average value, and in the satisfaction objective the goal is to maximize the probability of runs such that the limit-average value stays above a given vector. We show that under the expectation objective, in contrast to the single-objective case, both randomization and memory are necessary for strategies, and that finite-memory randomized strategies are sufficient. Under the satisfaction objective, in contrast to the single-objective case, infinite memory is necessary for strategies, and that randomized memoryless strategies are sufficient for ε ε ε-approximation, for all ε > 0 ε > 0 ε > 0. We further prove that the decision problems for both expectation and satisfaction objectives can be solved in polynomial time and the trade-off curve (Pareto curve) can be ε ε ε-approximated in time polynomial in the size of the MDP and

Playing Stochastic Games Precisely

“…Similar concepts for a strategy σ of the Spoiler are defined analogously. In this paper we use an alternative formulation of strategy [1] that generalises the concept of strategy automata [6].…”

“…Strategies are expressed as strategy automata [6,1] that consist of-i) a set of memory elements, ii) a memory update function that specifies how memory is updated as the transitions occur in the game arena, and iii) a next move function that specifies a distribution over the successors of game state, depending on the memory element. Memory update functions in strategy automata can be either deterministic or stochastic [1].…”

“…Memory update functions in strategy automata can be either deterministic or stochastic [1]. We show that the choice of how the memory is updated drastically influences the size of memory required.…”

“…The precise value problem studied here is a special case of multiobjective optimisation, where a player strives to fulfill several (in our case two) objectives at once, each with a certain minimum probability. Multi-objective optimisation has been studied for Markov decision processes with discounted rewards [4], long-run average rewards [1], as well as reachability and ω-regular objectives [7]; however, none of these works consider multi-player optimisation.…”

On Stochastic Games with Multiple Objectives

“…For non-stochastic games, multi-dimensional objectives have been considered in [6,23]. For MDPs, multiple discounted objectives [5], long-run objectives [2], ω-regular objectives [9] and total rewards [12] have been analysed. The objectives that we study in this paper are a special case of branching time temporal logics for stochastic games [3,1].…”