Distribution-based objectives for Markov Decision Processes

Abstract

We consider distribution-based objectives for Markov Decision Processes (MDP). This class of objectives gives rise to an interesting trade-off between full and partial information. As in full observation, the strategy in the MDP can depend on the state of the system, but similar to partial information, the strategy needs to account for all the states at the same time.In this paper, we focus on two safety problems that arise naturally in this context, namely, existential and universal safety. Given an MDP A and a closed and convex polytope H of probability distributions over the states of A, the existential safety problem asks whether there exists some distribution ∆ in H and a strategy of A, such that starting from ∆ and repeatedly applying this strategy keeps the distribution forever in H . The universal safety problem asks whether for all distributions in H , there exists such a strategy of A which keeps the distribution forever in H . We prove that both problems are decidable, with tight complexity bounds: we show that existential safety is PTIME-complete, while universal safety is co-NP-complete.Further, we compare these results with existential and universal safety problems for Rabin's probabilistic finite-state automata (PFA), the subclass of Partially Observable MDPs which have zero observation. Compared to MDPs, strategies of PFAs are not statedependent. In sharp contrast to the PTIME result, we show that existential safety for PFAs is undecidable, with H having closed and open boundaries. On the other hand, it turns out that the universal safety for PFAs is decidable in EXPTIME, with a co-NP lower bound. Finally, we show that an alternate representation of the input polytope allows us to improve the complexity of universal safety for MDPs and PFAs.

Stochastic Games with Synchronizing Objectives

“…The global state of the game is a distribution over the local states of the processes, and the specification describes which sequences of distributions are winning. The distributions can be discrete [AAGT12,CFO20] or continuous [KVAK10,AGV18]. The control may be applied uniformly, independently of the local state of each process, as in non-deterministic [BDGG17], and probabilistic automata [CFO20], or it may depend on the local history of states, as in Markov decision processes (MDPs) [AGV18,DMS19].…”

“…The distributions can be discrete [AAGT12,CFO20] or continuous [KVAK10,AGV18]. The control may be applied uniformly, independently of the local state of each process, as in non-deterministic [BDGG17], and probabilistic automata [CFO20], or it may depend on the local history of states, as in Markov decision processes (MDPs) [AGV18,DMS19]. In both cases imperfect information arises: either because the control is global, thus not aware of the local state of individual processes, or because the control is local, thus not aware of the global states on which the specification is defined.…”

“…The works on (discrete) parameterized control considered finitary synchronization objectives (reachability of a synchronized distribution), either with an adversary [BDGG17], or with stochasticity [CFO20], but not with both. With continuous distributions, the central model that has been studied is MDPs, thus with stochasticity but no adversary, either for finitary [AGV18] or infinitary objectives [AAGT12,DMS19]. The solution of the control problem studied in this paper is known for MDPs [DMS19].…”

“…Many systems in natural computing exhibit several instances of the same anonymous process (without pre-defined identity or hierarchy), from particle physics to flock of birds. Examples of biological systems such as yeast [BDGG17,AGV18], and simple chemical systems [KVAK10] illustrate the synthesis applications of this model. The same principle underlies synthetic biology where a local control program is executed in every instance of the process [EL00, NDS + 16].…”

Bounds for Synchronizing Markov Decision Processes

“…In the more recent distribution-based semantics, the outcome of a stochastic process is a sequence of distributions over states [3,19]. This alternative semantics has received some attention recently for theoretical analysis of probabilistic bisimulation [17] and is adequate to describe large populations of agents [14,10] with applications in system biology [19,1]. The behaviour of an agent is modeled as an MDP with some state space Q, and a large population of identical agents is described by a (continuous) distribution d : Q → [0, 1] that gives the fraction d(q) of agents in the population that are in each state q ∈ Q.…”