Solving Partially Observable Stochastic Games with Public Observations

Abstract

In many real-world problems, there is a dynamic interaction between competitive agents. Partially observable stochastic games (POSGs) are among the most general formal models that capture such dynamic scenarios. The model captures stochastic events, partial information of players about the environment, and the scenario does not have a fixed horizon. Solving POSGs in the most general setting is intractable.Therefore, the research has been focused on subclasses of POSGs that have a value of the game and admit designing (approximate) optimal algorithms. We propose such a subclass for two-player zero-sum games with discounted-sum objective function—POSGs with public observations (POPOSGs)—where each player is able to reconstruct beliefs of the other player over the unobserved states. Our results include: (1) theoretical analysis of PO-POSGs and their value functions showing convexity (concavity) in beliefs of maximizing (minimizing) player, (2) a novel algorithm for approximating the value of the game, and (3) a practical demonstration of scalability of our algorithm. Experimental results show that our algorithm can closely approximate the value of non-trivial games with hundreds of states.

Heuristic Search Value Iteration for Zero-Sum Stochastic Games

“…c) POSGs: As mentioned in Sec. II, HSVI has been first directly applied to particular zero-sum (two-player) partially observable stochastic games (zs-POSGs): with one-sided partial observability [11] and with public observability [12]. These problems allow (i) not having to deal with nested beliefs, and (ii) reasoning about beliefs such that the optimal value function is convex/concave in belief-space, which allows defining generalizing value function approximators (upperand lower-bounds) as in POMDPs.…”

“…Recent works addressed particular cases of discounted (twoplayer) zero-sum partially observable stochastic games (zs-POSGs) using heuristic search: Horák et al [11] considered One-Sided POSGs, i.e., the case where one player has access to the system state, plus the action and observation of the other player, and Horák and Bošanský [12] considered POSGs with public observations, i.e., the case where each Player i knows his own private state s i and both players receive the same public observations of each player's private state, so that they have common knowledge of Player −i's belief over Player i's private state. Moving from MDPs and POMDPs (as in Smith's work 2007) to these settings requires changes to the algorithm that make a different approach necessary to theoretically analyze the finite-time convergence.…”

“…1 Fct HSVI ( ) Comparison with HSVI for POMDPs and POSGs: Note that the finite branching factor, due to the finite number of reachable states and despite the infinite set of actions, allows deriving a version of HSVI with the same definition of the excess function and the same convergence result (including the bound) as HSVI for POMDPs [24]. In the contrary, the potentially infinite branching factor in POSGs considered by [11] and [12] requires adding a term to the excess function, which induces an increased δ max , and relying on the Lipschitzcontinuity of the optimal value function to prove finite-time convergence to an -optimal solution. 5…”

“…Such games can serve for instance in robust control [9], or in solving or analyzing Real-Time Strategy games (e.g., µRTS [10]). While particular partially observable SGs, hence more general problems, have been solved through heuristic search [11,12] (see next section), this work fills a gap by proposing and studying a simpler solution that is better suited to the fully observable case.…”

Compact Representation of Value Function in Partially Observable Stochastic Games

“…When both players have partial information about the environment, the players may need to reason not only about their belief over possible states, but also about the belief the opponent has over the possible states, beliefs over beliefs, and so on. Restricting to subclasses of POSGs where this issue of nested beliefs does not arise allows us to design and implement algorithms that are guaranteed to converge to optimal strategies [Horák et al, 2017;Horák and Bošanský, 2019]. However, the scalability of the current algorithms even for this case is limited.…”