Strategy Complexity of Finite-Horizon Markov Decision Processes and Simple Stochastic Games

Abstract

Markov decision processes (MDPs) and simple stochastic games (SSGs) provide a rich mathematical framework to study many important problems related to probabilistic systems. MDPs and SSGs with finite-horizon objectives, where the goal is to maximize the probability to reach a target state in a given finite time, is a classical and well-studied problem. In this work we consider the strategy complexity of finite-horizon MDPs and SSGs. We show that for all ǫ > 0, the natural class of counter-based strategies require at most log log( 1 ǫ ) + n + 1 memory states, and memory of size Ω(log log( 1 ǫ ) + n) is required, for ǫ-optimality, where n is the number of states of the MDP (resp. SSG). Thus our bounds are asymptotically optimal. We then study the periodic property of optimal strategies, and show a sub-exponential lower bound on the period for optimal strategies.

Game of Life-like Opinion Dynamics: Generalizing the Underpopulation Rule

“…The underpopulation opinion dynamics considered in [14] is a noticeable example of a (family of) local threshold-based opinion dynamics, which corresponds to having, for any u ∈ V, θ + (u) = c + and θ − (u) = c − for some pair of constants c + , c − ∈ N. The underpopulation dynamics ruled by the constants c + and c − will be denoted as…”

“…Goles and Olivos [13] proposed single-threshold deterministic opinion dynamics for weighted graphs in which every node at each step is either in state 0 or in state 1, and at the next step, it obtains state 1 if and only if the sum of the products between the current state of each of its k neighbors with the weight of the edge connecting the neighbor to the individual is at least θ(k) for a given function θ defined on the node set. In [14], a couple of deterministic opinion dynamics models, the overpopulation and theunderpopulation rules, that still work with binary node states are defined for unsigned graphs as a simplification of the Game-of-Life rules [15]; according to the underpopulation rule, nodes' opinion changes are controlled by two constant values, c + and c − . A node gets a positive opinion if either its opinion is already positive and at least c + neighbors have a positive opinion, or its opinion is negative and at least c − neighbors have a positive opinion.…”

“…In [43], it is proved a tight bound Θ(n 2 ) on the number of configurations met by an unsigned undirected graph evolving according to a deterministic majority dynamics (in which a node changes its opinion if and only if the majority of its neighbors has the other opinion) starting from any initial configuration. In [14], a polynomial bound in the graph size on the number of configurations met by an unsigned undirected graph evolving according to an underpopulation rule and starting at any initial opinion configuration is provided. In [16], a similar polynomial bound is first proved for unsigned undirected graphs evolving according to any local threshold-based dynamics; then, the bound is proved when the graph is signed and undirected, and it evolves according to an extension of the deterministic majority dynamics to signed graphs.…”