Determinacy in Stochastic Games with Unbounded Payoff Functions

Abstract

We consider infinite-state turn-based stochastic games of two players, and , who aim at maximizing and minimizing the expected total reward accumulated along a run, respectively. Since the total accumulated reward is unbounded, the determinacy of such games cannot be deduced directly from Martin's determinacy result for Blackwell games. Nevertheless, we show that these games are determined both for unrestricted (i.e., history-dependent and randomized) strategies and deterministic strategies, and the equilibrium value is the same. Further, we show that these games are generally not determined for memoryless strategies. Then, we consider a subclass of -finitely-branching games and show that they are determined for all of the considered strategy types, where the equilibrium value is always the same. We also examine the existence and type of (ε-)optimal strategies for both players. IntroductionTurn-based stochastic games of two players are a standard model of discrete systems that exhibit both non-deterministic and randomized choice. One player (called or Max in this paper) corresponds to the controller who wishes to achieve/maximize some desirable property of the system, and the other player (called or Min) models the environment which aims at spoiling the property. Randomized choice is used to model events such as system failures, bit-flips, or coin-tossing in randomized algorithms.Technically, a turn-based stochastic game (SG) is defined as a directed graph where every vertex is either stochastic or belongs to one of the two players. Further, there is a fixed probability distribution over the outgoing transitions of every stochastic vertex. A play of the game is initiated by putting a token on some vertex. Then, the token is moved from vertex to vertex by the players or randomly. A strategy specifies how a player should play. In general, a strategy may depend on the sequence of vertices visited so far (we say that the strategy is history-dependent (H)), and it may specify a probability distribution over the outgoing transitions of the currently visited vertex rather than a single outgoing transtion (we say that the strategy is randomized (R)). Strategies that do not depend on the history of a play are called memoryless (M), and strategies that do not randomize (i.e., select a single outgoing transition) are called determinisctic (D). Thus, we obtain the MD, MR, HD, and HR strategy classes, where HR are unrestricted strategies and MD are the most restricted memoryless deterministic strategies.

Efficient Analysis of VASS Termination Complexity

“…For every initial configuration 𝑝v, the players 𝐴 and 𝐷 aim at minimizing and maximizing the length of a computation initiated in 𝑝v, respectively. By applying standard arguments (see, e.g., [5]), we obtain…”

“…We also show that if L ∉ G 𝑘 , then L (𝑛) ≥ 𝐹 𝑘+1 ( ⌊𝑛/𝑐⌋) for all 𝑛 ∈ N and some constant 𝑐 depending only on A, i.e., the growth of L indeed reaches 4 the next level G 𝑘+1 . Due to this gap, the growth of L for a terminating 5 A is reasonably characterized by the least 𝜆 ∈ N such that L ∈ G 𝜆 (it follows already from the results of [19] 2 Actually, this result is obtained for a subclass of probabilistic VASS with tree-like MEC decomposition. 3 The standard definition of fast-growing hierarchy is simpler and sets…”

“…A computation initiated in a configuration 𝑝v is then determined by two players, 𝐴 (angel) and 𝐷 (daemon) who resolve nondeterminism in configurations with angelic and demonic control states, aiming at minimizing and maximizing the length of the computation, respectively. By applying standard results about games with accumulated reward objectives (see, e.g., [5]), we obtain that for every initial configuration 𝑝v there exists a unique termination value Tval(𝑝v) ∈ N ∞ defined by the equality where 𝜎 and 𝜋 range over the strategies of the players 𝐴 and 𝐷, and |Comp 𝜎,𝜋 (𝑝v)| is the length of the maximal computation obtained by applying the strategies 𝜎 and 𝜋 in 𝑝v. The termination complexity is a function L : N → N ∞ assigning to every 𝑛 ∈ N the maximal Tval(𝑝v), where 𝑝 ∈ 𝑄 and v ≤ (𝑛, .…”

“…4 Note that according to our result, it cannot happen that L ∉ G 𝑘 and L ≤ 𝐹 (𝑔 (𝑛) ) 𝑘 (𝑛) for some sublinear function 𝑔 (𝑛) such as √ 𝑛 or log(𝑛). 5 A VASS A is terminating if L (𝑛) < ∞ for all 𝑛 ∈ N.…”

Solving Stochastic Büchi Games on Infinite Arenas with a Finite Attractor

“…Remark (On randomized memoryless strategies). It is known that, if one considers mixed, aka randomized, strategies, generalized Büchi objectives on G admit memoryless, aka deterministic, winning strategies [20]. Note that our σ W is finite-memory (and not randomized).…”