The Asymptotic Theory of Stochastic Games

Abstract

We study two person, zero sum stochastic games. We prove that limn→∞{Vn/n} = limr→0rV(r), where Vn is the value of the n-stage game and V(r) is the value of the infinite-stage game with payoffs discounted at interest rate r > 0. We also show that V(r) may be expanded as a Laurent series in a fractional power of r. This expansion is valid for small positive r. A similar expansion exists for optimal strategies. Our main proof is an application of Tarski's principle for real closed fields.

Computing uniformly optimal strategies in two-player stochastic games

“…A key result that we use in our construction is that the function that assigns to every discount factor the discounted value at a given state has a representation as a Taylor series in a fractional power of λ in a neighborhood of 0 (Bewley and Kohlberg, 1976). Such a function is called a Puiseux function.…”

“…This existence result was later generalized to the existence of a stationary discounted equilibrium in multi-player games (Fink, 1964). Bewley and Kohlberg (1976) proved that the limit of the discounted values exists. Mertens and Neyman (1981) proved the existence of uniform ε-optimal strategies in two-player zero-sum games: for every ε > 0 each of the two players has a strategy that guarantees the discounted value, up to ε, for every discount factor sufficiently close to 0.…”

“…This algorithm relies on the following insight. By Bewley and Kohlberg (1976), the function λ → v λ that assigns to each λ the value of the λ-discounted game, is a semi-algebraic function of λ. It therefore can be expressed as a Taylor series in fractional powers of λ, and is monotonic, in a neighborhood of λ = 0.…”

“…As mentioned before, identifying the uniform value amounts to finding the limit v of this semi-algebraic function. Relying on Bewley and Kohlberg (1976), Chatterjee, Majumdar and Henzinger note that, for a given α, determining whether v > α is equivalent to finding the truth value of a sentence in the theory of real-closed fields. Tarski's quantifier elimination algorithm (or Basu, 1999, see also Basu, Pollack and Roy, 2003) can be used to compute this truth value.…”

Perturbations of Markov Chains with Applications to Stochastic Games

“…Puiseux series were introduced to the study of stochastic games by Bewley and Kohlberg (1976). Since Puiseux series form a real closed field, they proved to be a useful tool in analyzing asymptotic properties of discounted stochastic games.…”