Value Iteration for Long-Run Average Reward in Markov Decision Processes

Abstract

Markov decision processes (MDPs) are standard models for probabilistic systems with non-deterministic behaviours. Long-run average rewards provide a mathematically elegant formalism for expressing long term performance. Value iteration (VI) is one of the simplest and most efficient algorithmic approaches to MDPs with other properties, such as reachability objectives. Unfortunately, a naive extension of VI does not work for MDPs with long-run average rewards, as there is no known stopping criterion. In this work our contributions are threefold.(1) We refute a conjecture related to stopping criteria for MDPs with longrun average rewards. (2) We present two practical algorithms for MDPs with long-run average rewards based on VI. First, we show that a combination of applying VI locally for each maximal end-component (MEC) and VI for reachability objectives can provide approximation guarantees. Second, extending the above approach with a simulation-guided on-demand variant of VI, we present an anytime algorithm that is able to deal with very large models. (3) Finally, we present experimental results showing that our methods significantly outperform the standard approaches on several benchmarks.

Conditional Value-at-Risk for Reachability and Mean Payoff in Markov Decision Processes

“…When dealing with Markov Chains in queries, we only consider finite state sets. 2 In other words, each action is associated with exactly one state.…”

“…For the MDP case, recall that simple expectation maximization of mean payoff can be reduced to weighted reachability [2] and deterministic, memoryless strategies are optimal [31]. Yet, solving a conjunctive query involving either VaR or CVaR needs more powerful strategies than in the weighted reachability case of Thm.…”

“…This construction intuitively collapses each MEC into a single representative state and adapts the transitions accordingly. Since in each MEC every state can be reached from any other with probability 1, the MEC quotient preserves many infinite horizon properties like (weighted) reachability [2,19,31]. More precisely, queries can easily be transformed so that they are satisfiable on the MEC quotient if and only if the original query is satisfiable.…”

“…(4) VaR-consistent split for j ∈ [d]: x j s = x s for s ∈ T j < x j s ≤ x s for s ∈ T j = Probability-consistent split for j ∈ [d]: CVaR and expectation satisfaction for j ∈ [d]:s ∈T j ≤ x s · r(s) ≥ p j · c j s ∈Tx s · r j (s) ≥ e j LP used to decide multi-dimensional weighted reachability queries given a guess t of VaR p j . Equalities(2) and(3)are as in Fig. 4, T j ∼ := {s ∈ T | r j (s) ∼ t j }, ∼∈ {<, =, ≤}.…”

Robust Solution Approach for the Dynamic and Stochastic Vehicle Routing Problem

“…Usually, the dynamic and stochastic VRP is modelled either as a Markov decision process or as a stochastic program [1]. An MDP consists of a finite set of states, a finite set of actions, representing the nondeterministic choices, and a transition function that given a state and an action provides the probability distribution over the successor states [2]. Differently, SP determines a feasible solution for all possible outcomes [3].…”