A Pumping Algorithm for Ergodic Stochastic Mean Payoff Games with Perfect Information

Abstract

Abstract. We consider two-person zero-sum stochastic mean payoff games with perfect information, or BWR-games, given by a digraph G = (V = VB ∪ VW ∪ VR, E), with local rewards r : E → R, and three types of vertices: black VB, white VW , and random VR. The game is played by two players, White and Black: When the play is at a white (black) vertex v, White (Black) selects an outgoing arc (v, u). When the play is at a random vertex v, a vertex u is picked with the given probability p (v, u). In all cases, Black pays White the value r(v, u). The play continues forever, and White aims to maximize (Black aims to minimize) the limiting mean (that is, average) payoff. It was recently shown in [BEGM09a] that BWR-games are polynomially equivalent with the classical Gillette games, which include many well-known subclasses, such as cyclic games, simple stochastic games (SSG), stochastic parity games, and Markov decision processes. In this paper, we give a new algorithm for solving BWR-games in the ergodic case, that is when the optimal values do not depend on the initial position. Our algorithm solves a BWR-game by reducing it, using a potential transformation, to a canonical form in which the value and optimal strategies of both players are obvious for every initial position, since a locally optimal move in it is optimal in the whole game. We show that this algorithm is pseudo-polynomial when the number of random nodes is constant. We also provide an almost matching lower bound on its running time, and show that this bound holds for a wider class of algorithms. Our preliminary experiments with the algorithm indicate that it behaves much better than its estimated worst-case running time. This and the fact that every iteration is simple and can be implemented in a distributed way makes it a good candidate for practical applications.

Approximation Schemes for Stochastic Mean Payoff Games with Perfect Information and Few Random Positions

“…Since BWR-games with a constant number of random positions admit a pseudopolynomial algorithm, as was recently shown [5,6], we obtain the following results. …”

“…There are various improvements with smaller dependence on k [9,15,20,23] (note that even though BWR-games are polynomially reducible to simple stochastic games, under this reduction the number of random positions does not stay constant, but is only polynomially bounded in n, even if the original BWRgame had a constant number of random positions). Recently, a pseudo-polynomial algorithm was given for BWR-games with a constant number of random positions and polynomial common denominator of transition probabilities, but under the assumption that the game is ergodic (that is, the value does not depend on the ini-tial position) [5]. Then, this result was extended for the non-ergodic case [6]; see also [4].…”

Stochastic Mean Payoff Games: Smoothed Analysis and Approximation Schemes

“…(Even though BWR-games are polynomially reducible to simple stochastic games, under this reduction the number k of random vertices becomes a polynomial in n, even if the original BWR-game has a constant number of random vertices.) Recently, a pseudo-polynomial algorithm was given for BWR-games with a constant number of random vertices and polynomial common denominator of transition probabilities, but under the assumption that the game is ergodic [8]. However, the existence of a similar algorithm for the non-ergodic case or a non-constant number of random vertices remains open, as the approach by Boros et al [8] does not seem to generalize to these cases.…”

Strategy recovery for stochastic mean payoff games

“…Notice that our β-recurrent games are ergodic in the sense of [BEGM10]. The complexity of ergodic games has been settled in a recent work [CIJ14a] (see the full version [CIJ14b]), however we need for our reduction the extra properties of β-recurrent games.…”

“…The complexity of ergodic games has been settled in a recent work [CIJ14a] (see the full version [CIJ14b]), however we need for our reduction the extra properties of β-recurrent games. Interestingly, the definition of ergodic in [CIJ14a] is more restrictive than that in [BEGM10], and, in particular, in this stronger sense, a β-recurrent game may not be ergodic, nor an ergodic game needs to be β-recurrent. Lemma 3.…”