Reasoning about MDPs as Transformers of Probability Distributions

Korthikanti, Vijay Anand; Viswanathan, Mahesh; Agha, Gul; Kwon, Young‐Min

doi:10.1109/qest.2010.35

Cited by 29 publications

(55 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These specifications are undoubtedly interesting since they are ubiquitous; however there are many interesting and intuitive properties that they can not express, in particular, considerations about the dynamical evolution of state probabilities are either convoluted or impossible to state in these frameworks, as pointed in e.g. [15,13,7,1].…”

Section: Introductionmentioning

confidence: 99%

Decidability of Approximate Skolem Problem and Applications to Logical Verification of Dynamical Properties of Markov Chains

Biscaia

Henriques

Mateus

2014

ACM Trans. Comput. Logic

View full text Add to dashboard Cite

When studying probabilistic dynamical systems, temporal logic has typically been used to reason about path properties. Recently, there has been some interest in reasoning about the dynamical evolution of state probabilities of these systems. In this paper we show that verifying linear temporal properties concerning the state evolution induced by a Markov chain is equivalent to the decidability of the Skolem problem -a long standing open problem in Number Theory. However, from a practical point of view, usually it is enough to check properties up to some acceptable error bound . We show that an approximate version of Skolem problem is decidable, and that it can be applied to verify, up to arbitrarily small , linear temporal properties of the state evolution induced by a Markov chain.

show abstract

Section: Introductionmentioning

confidence: 99%

Decidability of Approximate Skolem Problem and Applications to Logical Verification of Dynamical Properties of Markov Chains

Biscaia

Henriques

Mateus

2014

ACM Trans. Comput. Logic

View full text Add to dashboard Cite

show abstract

“…On the contrary, in this work, the control objective is to achieve a given steadystate distribution. In a recent line of work [3,6], the authors consider transient properties of MDP viewed as transformers of probability distributions. Compared to that setting, we are interested rather in long run properties.…”

Section: Introductionmentioning

confidence: 99%

The Steady-State Control Problem for Markov Decision Processes

Akshay¹,

Bertrand²,

Haddad³

et al. 2013

Quantitative Evaluation of Systems

View full text Add to dashboard Cite

Abstract. This paper addresses a control problem for probabilistic models in the setting of Markov decision processes (MDP). We are interested in the steady-state control problem which asks, given an ergodic MDP M and a distribution δ goal , whether there exists a (history-dependent randomized) policy π π π ensuring that the steady-state distribution of M under π π π is exactly δ goal . We first show that stationary randomized policies suffice to achieve a given steady-state distribution. Then we infer that the steady-state control problem is decidable for MDP, and can be represented as a linear program which is solvable in PTIME. This decidability result extends to labeled MDP (LMDP) where the objective is a steady-state distribution on labels carried by the states, and we provide a PSPACE algorithm. We also show that a related steady-state language inclusion problem is decidable in EXPTIME for LMDP. Finally, we prove that if we consider MDP under partial observation (POMDP), the steady-state control problem becomes undecidable.

show abstract

“…Recently, an alternative semantics for probabilistic automata has been proposed, with applications in sensor networks, queuing theory, and dynamical systems [9,8,5]. In this new semantics, a run over an input word is the sequence of probability distributions produced by the automaton.…”

Section: Introductionmentioning

confidence: 99%

“…The space of probability distributions (which is a subset of [0,1] n ) is partitioned into regions defined by linear predicates, and classical acceptance conditions are used to define accepting sequences of regions. It is known that reachability of a region is undecidable for linear predicates, and that it becomes decidable for a class of qualitative predicates which essentially constrain only the support of the probability distributions [8].…”

Section: Introductionmentioning

confidence: 99%

Infinite Synchronizing Words for Probabilistic Automata

Doyen¹,

Massart²,

Shirmohammadi³

2011

Mathematical Foundations of Computer Science 2011

View full text Add to dashboard Cite

Abstract. Probabilistic automata are finite-state automata where the transitions are chosen according to fixed probability distributions. We consider a semantics where on an input word the automaton produces a sequence of probability distributions over states. An infinite word is accepted if the produced sequence is synchronizing, i.e. the sequence of the highest probability in the distributions tends to 1. We show that this semantics generalizes the classical notion of synchronizing words for deterministic automata. We consider the emptiness problem, which asks whether some word is accepted by a given probabilistic automaton, and the universality problem, which asks whether all words are accepted. We provide reductions to establish the PSPACE-completeness of the two problems.

show abstract

Reasoning about MDPs as Transformers of Probability Distributions

Cited by 29 publications

References 20 publications

Decidability of Approximate Skolem Problem and Applications to Logical Verification of Dynamical Properties of Markov Chains

Decidability of Approximate Skolem Problem and Applications to Logical Verification of Dynamical Properties of Markov Chains

The Steady-State Control Problem for Markov Decision Processes

Infinite Synchronizing Words for Probabilistic Automata

Contact Info

Product

Resources

About