Value Iteration for Simple Stochastic Games: Stopping Criterion and Learning Algorithm

Abstract

Simple stochastic games can be solved by value iteration (VI), which yields a sequence of under-approximations of the value of the game. This sequence is guaranteed to converge to the value only in the limit. Since no stopping criterion is known, this technique does not provide any guarantees on its results. We provide the first stopping criterion for VI on simple stochastic games. It is achieved by additionally computing a convergent sequence of over-approximations of the value, relying on an analysis of the game graph. Consequently, VI becomes an anytime algorithm returning the approximation of the value and the current error bound. As another consequence, we can provide a simulationbased asynchronous VI algorithm, which yields the same guarantees, but without necessarily exploring the whole game graph. ⋆ This research was funded in part by the Studienstiftung des deutschen Volkes project "Formal methods for analysis of attack-defence diagrams", the Czech Science Foundation grant No. 18-11193S, and the German Research Foundation (DFG) project KR 4890/2-1 "Statistical Unbounded Verification". 1 Formally, the problem is to decide, for a given p ∈ [0, 1] whether Maximizer has a strategy ensuring probability at least p to reach the target.

PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games

“…2. The transition from algorithms for MDP to SG is possible via extending the over-approximating value iteration from MDP [BCC+14] to SG by [KKKW18].…”

“…[WT16] combines the special case of almost-sure satisfaction of a specification with optimizing quantitative objectives. We use techniques of [KKKW18], which however assumes access to the transition probabilities.…”

“…the value function. Since it is difficult to give a convergence criterion, the approach of bounded value iteration (BVI, also called interval iteration) was developed for MDP [BCC+14,HM17] and SG [KKKW18]. Beside the underapproximation, it also updates an over-approximation according to the Bellman equations.…”

“…Note that, to stay in an EC in an SG, the two players would have to cooperate, since it depends on the pair of strategies. To take into account the adversarial behaviour of the players, it is also relevant to look at a subclass of ECs, the so called simple end components, introduced in [KKKW18].…”

“…Algorithm 1 rephrases that of [KKKW18] and describes the general structure of all bounded value iteration algorithms that are relevant for this paper. We discuss it here since all our improvements refer to functions (in capitalized font) in it.…”